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ABSTRACT 

This study is one of three designed to evaluate the 
effectiveness of the Onified Science and Mathematics for Elementary 
Schools (OSMES) program.' The student study examined the program's 
impact on students • development of problem-solving abilities and on 
their attitudes. Conditions affecting the program's impact were 
identified, including the amount and type, of USHES experience 
students received, their schools, gra'Se levels, and scholastic 
achievement levels. Instruments were developed and administered to a 
small number of carefully studied schools. The OSHES program did not 
appear to have a statistically significant effect oni the 
problem-solving ability of the students. However, the students did^ 
seem to enjoy doing OSHES work. . (BB) ^ ' 
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Exectrtiv^ Summary of the USMES Student, 




- The USMES. Student Study examined a niomber of ^effects of the USMES 
program on students- .The specific issues selected, for study were sugge^sted 
by a substantial body of past research and by. the USMES staff Vs desire to 
provide developers and users of -USMES units with useful new information 
on the program's impact under a variety of conditions, as well ^-as with 
prototype diagnostic instruments. 

Methods ^ ^ ' • ^ . 

To measure student problem-solving .skills and strategies the USMES- * 
staff developed a consiimer product research problem (called the Pencil ' 
Problem) that would enable them to look at a few key indicators: ' ' 

1. Does the student consider the possible importance of many;' . 
factors, like product performance, durability, and cost? ; 

2. DoeSyJihe student - systematically perform experiments and col- 
\^ lect useful information? 

3. X^hen advising others about the product, does the student, 
present relevant information in effective ways? 



^Copies of the final report of the Student Study are availcLble> from the 
.USMES Office at Education Development Center. ' \ ' 
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The Pencil Problem asks students, in groups of three, to examine six brands 
of pencils and^ recbmmend whiph brand their principal* should buy. Students 
aire then c[uestioned about their approaches toward solving the problem and 
observed while, they try the pencils. A similar, though shorter, process is 
repeated using erasers. Students' verbal and written responses are recordeji 
^d later coded.' ^ . - . 

This consumer product research setting seemed to^offer a realistic 
sma^ll-scale. problem as well as a useful source of observations about the 
students-. The poten-ti;^l richness of observations suggested that, the instru- 
ment I might be suitable "fbjr dxagnos'toicT' use . by .USMES classroom teachers-. ^j:^ 



1 • • ■ ' ' • * 

. ^- - ' To- measxire student attitudes, the USMES staff develope'd a three-page 

multiple-choice Questionnaire on Attitudes Toward Real problem Solving (QAEPS) 

to 'assess three bro.ad types of 'student attitudes: 

1. * Attitudes toward working on real problems . and producing 

effective solutions . ' . 

2. Attitudes toward the group ^^teraction involved in real 
problem solving 

' ,,3.- Attitudes toward a number *of specific problem-solving 
activities. 

■ A - ■ ^ . • -. ■ ■ ■ .. 

The QARPS also included a section, only for students with USMES experience, 
covering their attitudes toward USMES as a school progrcim^ , 

'The student sample came from, twenty-four classes in four elementary 
schools that differ in size^and composition of student i)Odies, and in the 
nature of their surrbunding communities (see 'Table below). -All USMES classes 
aibove the first grade were included, as well as a n-omber of comparison 
classes that were not doing USMES^ at the' time. Classes chosen for compafi- 
'son were those, judg^ed , by school principals as most equivalent to the USMES 
classed in grade level, student 'ability level , and teacher's experience. 

• The QARPS wais administered to all students, in all saiaple classes; the 
Pencil Problem, was administered to a random sample of students in each class. 

i The USMES classes included in the study had used a' wide variety of 
published USMES .units as well as several IpcallJ-developed units. 

Three members of .the USMES research ^ staff "visited each school^during 
Apr il^ and May 1977 to ^administer the OARPS and the Pencil. Problem. A proce- 
dure-was devisecl to ensure that, .in^ administering the Pencil Problem to* a 
group of students they would not know.- what class the students had come from. 
This, helped rule out biased observation or conversational tone. 

Results -and Conclusions ) . "\ ' 

The mean scores -of the USMES students were* higher than those of the 
comparison students on virtually every major outcome. However, these dif- 
ferences were only statistically significant on some of the. outcomes. The 
effects bf USMES were generally small, compared to the effects^of School, 
Grade, and Achievement Level.- *^ - / - 
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CHARAqTERISTICS OF SCHOOLS 
(Alj? figures are approximate) 
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Because of the small^ and' uneven effects of different ainpupts of USMES 
experience, a linear regression analysis showed only moderate statistical 
significance for Amount of USMES on four of the six majoi^^iatcomes and ppor 
significance levels on the other two outcomes. One of the statistically 
-""^ significant regression results, was * negative; the other threq. were positive. 

-V, . . 

; The effects of Amount, of USMES were most prono^Inced in a few settings,. 
. such as School D. In that school-^" the 'comparison group scored the lowest of 
all four comparoTson groups on the thr^e. Pencil Problem outcomes. Yet, the 
USMES group which had received a substantial ^ount <Df_ USMES experience in 
School D scored the highest^ of all _ four, schools' "substantial USMES" groups 
on two of the three outcomes and second-highest on the other. Similar effects 
of Amount of USMES were found in the 6th grade (however, since many of the 6th* 
graders in the sample caume from School D, the 6th grade and School D results 
may bte related). ^ ^ - ^ . 

The motivational effects ot USMES did not seem to be so localized. The 
♦ responses to questions directed at students with USMES experience confirmed 
once again a repeated finding: ^ the great majority of USMES students enjoy 
USMES work, find it important, and would like to do more. This finding 
appeared to'^b^ true in all the schools and at all grade levels. 
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Several, factors seemed to account for the difference between motivational 
and problem- solving effects of USM^S. iizidently, "little" amounts of USMES 
^ (USME§ -experience in the past only, or less than 20 hours of USMES in the • 
current school year with no past experience) produced inconsistent results. 
Substcmtial amounts of USMES experience — including amounts greater than the* 
20 hours which defined "siibstantial" in the Student Study —may be necessary 
for consistent , significant problem7Solving results. 

The specific activities done in an USMES class may also be\^crucial. _Tihe. 

USMES School Study*^ found evidence that, opportunities for skill devel'opment 

^were often closely asspciated_ \vi:th the occurrence of se^rious investigations 

of data, "extended" discussions among students, and other recommended- USMES 

activities. Furthermore, in the Student Study the "siibstantial" Amount of 

USMES group appeared to have^'engaged in significantly higher levels of many 

recommended activities than^the "little" Amount of USMES group., and the 

"s^^fe^tantial" group produced the most consistently positive results. 

f' ^ 
In addition, the surprisingly large effects of School (Which xnclxi^ed 

some community differences in, the Student Study) , Grade, and Achievement Level 

suggested that -all of these factors made\ important contributions to tlie 

impact of USMES experience. ^ * ^ 

Finally, the influence of the school administration appeared to b(e cru- 
cial. ] In School B, USMES had^its most positiVe effect on attitudes, and in , 
School D USMES had its most positive -effect on prpblem-solving skills. In 
each school, these were the -outcomes that the principal of the school desired 
most to promote vsee the USME^ School Study report).^ The particular style ^ 
of" USMES chosen in each school emphasized the desired outcomes and was success- 
ful in improvin«j it. * , ' - . 



The results of thi^ study suggested €hat the choice of USMES" units (and 
the local development of units') should be -based on a careful corisideration ^ 
of the outcomes that would be most , valuable for a particular-school ^and class. 
Fxirthermore , it appeared that individual fetud^nts may ei^e'rience- a wide range 
of effects^of USMES experience. While virtually all students seem to enjoy 
USMES work, therei, Sre^ms to be considerable -variation from student to student 
in ju^t what is ^Learned. For consistent, significant results on problem- ' 
solving skills, the Student Study and the USMES School Study suggest ^that 
"regular" USMES units of some duration — with recommended USMES teaching methods 
-'-should-^ generally be^ preferred over abbreviated or haphazardly-created units. 

■ ■ ■ - L^- 

■ ■ ■ 

The QAPPS,and the- Pencil Problem seem to be use'Tul instfniments for 
diagnostic purposes. The QARPS is easily administered, easily -analyzed",' and 
shows satisfactory discrimination . » The Pencil Problem probe^ real-problem- 
solving abilities more "deeply tlian Tpencil and paper" tests, yet can be 
administered and analyzed in practical ways. ... r ^ * ' ^ 



A more^etailed. "summary of the' Student Study ' g findings be found 

on pp. ^I^t of ■ the' Stiident Study report. ^ . , 
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'^Available from the USMES oroject office at Education* Development Center^ 
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INTRODUCTION 



'The USMES Project conducted three studies in 1976-1977 to learn 
more about USMES students, schools, and resource teams. - The Student;^ 
Study examined the USMES program's impact on students' development of 
problem*- solving abilities and on their attitudes • A major goal of the 
Student Study was to identify conditions affecting the program's im- 
pact , including the amount and type o'f USMES experience students re- 
ceived, their schools, grade levels, and scholastic achievement levels. 

The general strategy of the s'tudent Study was to develop instru- 
ments for assessing key problem-solving abilities and attitudes, then 
to administer the instruments in a small niamber of carefully studii^ 
schools. This strategy allowed us to study a wide variety of students 
and USMES classes, at the same time that we collected fairly extensive 
observations about the particular schools'v communities, teachers, and 
class, activities. .It also enabled us to design* instruments that met 
2uiother important criterion— that they be suitable for diagnostic use 
by treachers and administrators. 

This report covers the major goals, methods, einc^'findings of the 
Student Study. Only findings on a few key ^outcomes causal factors 

are covered; because of serious time constraints we have not attempted 
to reach final conclusions but rather to summarize our impressions about 
the major findings. The report is intended for use by funders, school 
people, and other's interested in the impact of USMES on 'students. It is 
accompanied by brief technical reports on the' instr\aments used and, an 
^appendix showing detailed findings- 
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- '\ BJ^CKGROUND 

The specific issues selected "by. the USMES staff for attention dLn 
the Student Stu4y reflected -a number of .influences x A major influence 
Mas the subslStatial body of past research on USFlES .^-Ofie . summary of that 
research listed the following findings:* ^ 

• "In interviews with outside evaluators, teachers over- 

whelming]^ reported that USMES was fostering development 
of practrbal problem-solving abilities and social skills 
in their students. Fiirthennore, the administrators, 
teachers, emd students who jl^ere interviewed by Boston- L 
diversity personnel in 1975 \inanimously agreed that stu-^ 
zs enjoyed USMES and looked forward to using it, 

• " Although inconclusive results were obtained from the 

Playground Problem*-* and the Picnic ProblenT**, apparently 
because of limitations in the instrxanents themselves , Tne 
results from the Notebook Problem** showed dramatic gains 
in some Specific problem-solving skills for USMES students 
in contrast with much less positive results for comparison 
students • Further development of instruments to assess 
problem-solving- abilit-if»s is needed, and efforts in this 
direction are being made. 
-» 

• "Assessments of student achievement in reauiing, mathe- 

matical computation, and several other subject areas have 
foTind that USMES students were progressing at least as 
rSpidly as comparison students. In virtually every study 
where differences appeared between USMES and comparison 
. - groups, ^the trends favored the USMES group. Most of 'these 
. . differences vere small, but they are encouraging and war- y 
rant furthei investigation. ^ ^ 

• " Data gathered vith classroom observation scales has shown 

that 'JSMES s|:udents spend more time in self-directed, active, 
aiid creative behaviors thatn students in- comparison classes. 
USMES students also generate more ideas in both large and 
small group discussions. 



*See Chapter V of the fourth edition of the USMES Guide, ^DC, 1976, 
**Instrxunents developed at Boston University between 1971 and 1975 
.>to^ assess problem-solving skills for evaluations of USMES'. 



m "All of the available evidence on affective outcomes (from 
interviews ajid questionnaires) suggests that students 
find USMES challenging and inviting. Moreover , the pro- ^ 
gram may have significant positive effects on their ap- 
preciation for science." , ^ „ 

Taken together the findings of past research suggested that further re- 
search on student attitudes and prctolem- solving abilities might be es- 
pecially valuable.* 

Another major influence on the choice of issues was our desire to 
provide useful information to people involved in the development of USMES 
units. This group includes not only members of the USMES central sta'fff 
their funders, and their advisors, but also a large number of school 
teachers and administrators? trained USMES teachers and administrators 
have always been the primary soxirce of ideas for USMES published units / , 
emd they continue to develop additional units fpr local use. For the 
benefit of all these people involved in USMES development,, it also seemed 
important to le«m more about USMES *s impact on student attitudes and 
problcan-solyxng abilities, under a variety of conditions. 

In addition to providing developers and users of USMES with new in for- . 
mation, we W2uited to provide them with diagnostic -instruments on problem- 
solving abilities and attitudes, at leaist in prototype form. However, 
practical; constraints meSnt that we could only expect to develop one instru 
ment on /each outcome. Hence, o;ir research issues had t^ be measurable with 
a single attitude instrument and a single problem- solving instrument, both 
adaptable for use by others. 

While the choice c5f issues-' to be examined in the Student Study involved 
baleoicing many concerns, th^ decision to design the study around existing 
classes and to rely almost entirely on data collected near- the end of the 
school yecur was a less complicated one: we had no practical alternative. 
Time and budget constraints required us to study classes which were already 
committed to their 1976-1977 instructional programs before we contacted 
them. Furthermore, the time needed for instrument development prevented 
us from collecting research data with the instruments until the spring of 
1977. 

However, we believed that the limitations imposed by such a researph 
design would not prevent us from gathering useful informatior^ Tiecause we 
believed we could overcome many of the' limitations of the design. For ex- 
ample, we believed that we could locate groups \CLt students who were essen- 
^tially comparable except for their use (or non-use)' of USMES. Furthermore, 
we believed that we could collect information other than pre-test scores, 
(e.g., recent achievement test scores) which would bear upon the compara- 
bility of students receiving different amoxints tf USMES. Finally, we knew 



♦A^reanalysis by the USMES research staff of the Boston University study ^ 
of USMES and "basic skills" has shown that further research in this area * 
\Ls also needed. See Appendix I. 
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tJiat 'we^coiild' restrict our- interpretations 'of Student Study data in ac*^ 
cordance with the limitations imposed on the study. In general, we t^ok 
a position «du.ch we believe^ is "^imileu: to .that o£. Ddnald Campbell cuid\^' 

Julic^i Stanley when they wrote: - " . \ ^ 

> » ^ ■ . . ^ * 

"From the stcindpoint of ,the filial interpretation of cm * s 

eaqperimeixt . . • every experiment i's iniperfect. . ... (The 
experimenter) should seeklcfu't those artificicLL and . 
natural labpratoifi'es whipQ'^ovide^^'the best, opportuni- ' 
ties for control. But beyond that- he should go ahead . • 
, with the e^^riment and 'ij^^rpretation, fully aware of 

the points on which the r^fiiits are^equivofcal."*^ ^ • . : • 
' : ] ^ ' ' ' . ^ • , « ' ■ 

Otp:. decision to conduct the Student Study in only four schools, aind 
the s^ection of the particular schools in the ^tx2dy ,^ was dictated,;^y a 
number of requirements J We wcmted to -collect a relatively lairge ^ount 
of information, about the types o^f USME5 experience i-eceived by* each USMES 
Qla^S iij* the study, through site .visits,, teacher repprtsr- and interviews;, 
butr to dp so, jwe had to^res,trict the number of schools. We also wanted 
to work.withr schools where we would find students who had 'prior USMES ex- 
perience. A* wide range of student abilities, school - types , and surrounding 
communfties was also needed, to 'help identify conditions affecting the^ im- ^ 
pact of USMES on students. -Furthermore, logistical considerations demanded 
that we work with schcx^ls having a relatively laVge number of USMES classes 
and schools in which we could conduct a related USMES research study. Fi— 
nallyv because we would be asking schools for considerable cissistance and* 
because time was -«hort, we had to work with schools which were at least*^ 
moderately familiar^ to the USMES central office. We" Vpre fortunate that 
the schools we decided to study v^re all willing to work with us and were, 
in fact, extraordinarily helpful collaborators throughout the research. 



♦Campbell, dIt. & ^Starfley , J.G. Experimental and ^ quasi-experimental 
designs for researah. Chicago: Rand McNally & Co., 1^66. 
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^ Since protlextiT^^olving' skills- and strategies are a cj^tral concern of 
US WES/ a major fodus ,of the Student Study wais on these outcomes. There 
are serio\is problems in 'assessing students *1 problem-sowing abilities / 
however. One is the absence of cpmmercial 'instruments ror^ nieasuring the 
kinds of problem-solving abilities which USMES tries to foster • . Another , 
is thaJLength of time it takes to solve a reaiv prpbleni, USMES classes 
usually work for a number of w^eks on a singler probl^ and it is dif- 
ficult to imagine a group of students working iffeaningfully through all ^ 
the stages of real problem solving , in less thcui several hours. ^ To assess 
a student* s performance in alljfthe stages; of real pro5?lem solving would * 
require so mucfi time that only a few stude&its could be assessed . Yet ^ ' 
our desire to study students with a wide variety of USmIiS experiences 
{and competrable students with no USMES expe5ience) ^ meant that we had to 
assess a relatively Icurge ntainber o^,'^ students. > t 

To resolve . this dilemma , we decided to focus on only a few key indica- 
tors of "^students' problem-solving ^ abilities. Our decision* was based on 4£he 
belief that important skills and strategies used during at least' some stages 
of real problem * solving could be systematically observed in a reasonable 
amo\int of time. VOiile such -observations would not constitute 4r complete 
meeisure of real prctolem solving icompetence, they would provide useful infor- 
igat±on about the level .of students' ^problem- solving development. This view 
had been supported by the USMES Planning Congnittee^s Evaluation Panel, 
which% stated : 

"The Panel held hope for the feasibiSLity of developing 

measures of 'indicators.' Indicators were understood to 
meaJtv abilities or characteristics that j>resumably would, 
accompany the growth\of problem-solving ability. Such 
indicators can be lo^ed upon as desirable in themselves . 
— as well as indicators of problean-solviijg ability.''* , . . ] 

rftie problem-solving indicators on v*xich the Student^ Study focused . in- 
cluded the following: . M 

i '1. : While working on a ponsumer product research problem, 
does the Student consider the possible imjportsince cyf '% 
^ many factors, like the product's performance, durability, 
1 and cost, or does the student ^Dverlook/'many of the fajCtors? 



*Giudel±nes>^r 'Evaluation of USMES : Outcomes r EDC, 1975. 
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2. In the 'same sjettij^g^ does the student •'iry^%^ " 'r i'. - 

atically collect information, that^ is .xSL^ani 'and ^ 
useful^ > ' / ■• * s \ ^C'. . . • y 

J* When advising others abcjut iiie^ product investigated, ^ 
does the student present ^relevant in'forination in ef- ' 
fective ways? - v - ^ J 

These prdblem-solv^g indicators were selected because they seemed to 
'include outconie3, that wer^ of central coi^ern to USMES and^^e^e rea- 
sonably f eeisibleSN. to assess. . Furthermorev the consutDer product research , 
setting seemed to offer a realistic, small-scale, problem for .students 
as well as a rich source of dbservatifcns.>abput students:, ^^^^ riphness 
of potenticJ. observations cd.so made/them atjata^lif^ for^ use 
by USMES classroom teachers, . v v:*" ; 

Problem-solving skills .and stoat^gies Vere not^ only outcomes of 
concern in the Student StudyV The otiier-major /oCus o^ Study 
was student attitudes , because sc»ae * of ^tiie 'i^o^^ USMES goals 

involve irelatively, subtle changes in ^tt^itu4es<4^F6r example, USMES ex- 
perience is expected to 'foster a sense of co^giti'desice in students about 
their capacity, for finding .good solutions' ^X) Veal prob3ke?is and for imple- 
menting their solutions. ' Furthermore, USMES 'exp^erienc^' is expected to • 
promote ^a positive attitude toward working with qJ±l^^ students in real 
p'roblem solving. > . . ^ '^-^ ^ 

TO decide which attitudes- were most impifertaht for the Student Study, 
the USMES staff reviewed .a nuiiiber of instrWents" d6^^eloped for previous 
research on USMES and oh other curriculaL.,i In addition,, both/ the USMES 
staff and a Boston University USMES- evaluation ieain^.had comp^ 
reviews of several commic^-ally available ^attitudev^^ None of 

these reviews located an inStrxament which^se^ed yS^ cpver:^ an appropriate 
set" of attitudes* However, some questionnairj^ it^ems -i^^ re- 
search on the USMES program suggested topics that migfetjDe incor^ 
in a new q[uestionliaire.^ ■ ^ * ^ - ' . 

.On the basis of these"^ reviews , the USMES staff decided, to'^develop a 
student attitude questionnaire for , the- Student Study. It was to .assess 
three broad types of student attitude^: 

1. attitudes toward working O^real problei^ and producing " - 
effective solutions ^ / 

2. attitudes, toward' the group ij)£er action involved' in real 
prgbl^ solving 





^. attitudes/toward a number of specific probl em-sol viiig 
activities ^ . ■ , ..' 

In addition, 'the questionnaire was to^include a se'ction oJ^v'for students 
'with USMES expedience, covering their attitudes toward TJSMfiS. as a school 
program they had taken part in. ' 1 * 



Independent Variables . . * ^ - '/ 

^ / 

^ A ceixtral poncem in the Student Study was to examine the ixnpact of". 

varying amounts, of USMES eacperience* Consequent J^y , -we viewed "amount of 
-USMES" (bot^ the extent of a student's USMES experience this. year, if any, 

and whether or npt the student had prior USMES experience) as a key in- 
dependent veuriable.' After examining reports of actnxal USMESi'use this 

year, we decided to divide "^ainount USMES" into thre^ broad levels: 



1. 



'none' 



no USMES experience/' past or present 



2* "little": eithe? USMES experience in the past alone, 
•or up to 20 hours of USMES this year (but none in ifhe 



past) • 

* ■* , ' . • ^ 

3. "svfcstantial" : either USMES experience iri the pas,t as 

well as the present, or over 20, hours of pSMES this year 
(with or without past USMES) ^ , , 



^ Other' independent 'vferi£d)les given higfi priority for -study included 
school (i.e., the four aifferent schools in the study) and grade level. 
rChese two independent varieibJLes seemed important both for identifying 
conditions* affecting the impact of USMES and for statistical "control" - 
of the findings. ♦ . ^ ^ 

For the same reasons two additional independent variables were 
identified for study, but they were given. lower priority because of dif- 
ficulties involved in measuring them. These two variables were achieve- 
ment test scores and "type of USMES." . . 

Achievement test data were difficult to analyze, because the' fOTir 
schools of the Vtudy used, a number of different achievement tests, .admin- 
istered at different times of the year, and scored in different ways (e.g. 
local percentiles versus national grade-equivalents) . For our purposes , 
we could - not -justify cisking students and schools to"" submit to a special 
round of achievement testing just for the USMES Student Study; hence, we 
opted to use only achievement data the schools were routinely collecting 
and to analyze it cautiously. Ultimately, we only attempted to collect, 
achievement data for the relatively small number of students tested on 
.problem-solving indicators. ^ ■ . * . 

The independent variable, "type of USMES", " was intended to be a panel, 
rating (by USI4ES staff members) of USMES activities in the USMES clcisses 
of the study along several dimensiqns. Over the course of 'the past year, 
we experimented ^th' g number of possible rating dimensions , but none 
seemed simultaneously easy to code and quantitatively related* to even the 
most obvious student outcomes. Consequently , we '^veAtually decided 'not 
to.atten5>t a quantitative analysis of "type of- USMES," but rather to use 
the information about USMES class activiti^ as an^informcil aid iri inter- 
preting findings on other variables. ^ . 
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~ ■• INSTRDJffiNTS AND OTHER DATA SOURCES 

■ V. ' -The main ihstxuxnent for measuring problem-solving indicators was 

/ • ' .the Pencil Problem*, which was -developed for the Student Study. The , . 
> ' ^: Pencil Problem asks students, in grox5)s of three, to look over six ^ 

V brands of ^iencils and to recommend which bremd their principal should^ 
buy for students to use. Ihe students know that their .principal really 
will buy $10.00 worth of pencils based on their recommendations. Stu- 
dents are asked" several questions about their, ^^pproaches toward solving ; 
. - ^ ^ the^ problem and. the reasons for their choice, and they are systelnatipally 
obseirved while they try- the pencils . Students are also asked to look 
^ over five brands of erasers under similar conditions, and to tell how 
they would choose an eraser if they had time. Their responses to' the 
questions are tape-recorded, and they are also asked to make written 
recommendations to their principal about their choice of pencils. The 
verbal cind written responses are coded with checklists that cover the 
'essential phras^es of virtually all common responses. -Scores on a number 
of problem-solving indicators, such as considering most of th^ important y, 
information about each alternative, can'be derived from the codes. A ^ 
more complete description ^of the Pencil Problem is given in the. accompany-- 
ing Technical Report on the Pencil Problem: - \ ^ 

Students attitudes were assessed with the Questionnaire on Attitudes 
' ^ . toward Real Problem Solving (QARPS) , which was also developed for the^ 

student Study. The QARPS is a three-page , multip'le-choice. questionnaire 
" with two sections. The first section, with 20 items, is for both USMES . 
students > and students with no USMES experience'. The second section (on a 
separate page) is^^oniy for USMES students; it contains nine attitude items 
' . about -the USMES - program and a checklist about -experiences the students may 

have had while, doing USMES work. A more* complete description of the QARPS 
is contained in the- accompanying' Technical Report on the Questionnaxre 
on Attitudes Toward Real Problem Solving, , -i 

' Information on students V USI^ experience was\pbtained fromVseveral 
so\irces. For students in USMES classfes- this year, weekly teacher reports 
(or interviews with teachers who were unable, to send ' reports) provided in- 
formation abqut the amount , of time . students spent on USMES and the nature 
of their activities during USMES" time."^ The report and interview . forms 
'Were, developed on the basis of several years' experience wi^ USMES . teacher 
-'logs of class activities. ' The. questions' on the: forms were designed to pro - 
■ vide information on the critical aspects of th^ students' USMES' experience. 
• Sample^of the teacher report and interview forms are included in Appendix 
J. The amount of USMjIS experience a .student had this year was calculated 
(in hours) on the basis of each class's total USMES work for the year. 
In addition each school^Tprovided information about whether a student^ - 

• ■ ^ / <^ V . • : . . . ^ \ 

® *a3ie. Pencil Problem also includes a short "eraser problem*' as well as a 

ERIC , much" longer "pencil problem." 
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had done USMES work ;Ln past years • ,Thus, the research data inciuded the 
•axnoT&t of USMES experience this year (if any) 'for every class in o\ar 
saxi5)le ccnd also whether or not each stxident in the sample had done USMES 
in the past . ^ , . ^' ' ^ V 

Achievement test data came from' different tests in different schools. 
.TWO ^ schools used the Xova Tests of Sasic Skills, one used the SSA Assess- 
ment Survey Achievement Series^ Multilevel Edition, and one used the 

• Metropolit^ Achievement Tests. >riie only* common reporting format tor: 
achievement Scores^ among the four schools/ was national grade-equivalents 
We recognised that grade Tequiyalents are^ hazardous data in general^ and 
especially when they come from different tests. However/ we decided to , 

"^use the available grade -equivalent data^ and to interpret them cautiously^ 
rather than to ignore achievement levels^ entirely. - , 



• . SAMPLE' , 

The sainple of students in the Student Study came fr^m four very dif- 
ferent elemental schools. Some key characters tics of the four schools - 
are indicated in Table 1, As indicated in the table, the. schools differ 
not only in the size and; composition of their student Taodies , but also in 
the nature of the coiranunities in which they are l6ca^ed, 

,A11 USMES claisses in the four schools, above the first grade were 
included in* the Student Study. (Kindergarten and first grade classes were 
. not included, because our instruments were not appropriate for very young 
students .^^ In addition, a number of qompcurison classes. which were not 
doing USMES this year were included. Since we were primarily interested 
in USMEiS classes which*^ appeared from teacher reports to have done signif- 
icant 2unoTints (at least 20 hours) of USMES this year, we tried to include 
compaurison classes which were most equivalent to the relatively active 
USMES classes*. To select the comparison classes, we provided each prin- 
cipal with 'a list of the active USMES teachers and asked the principal^ to 
identify, for each of these classes, a comparison class which had not done 
.any "USMES this year and was similar in terms of grade levesl, ability leyel 
of students, cmd experience of .teacher. : Occasionally, there were no com- 
pa^rison classes availcible at appropriate grade levels, and in those cases, 
a class at the closest grade level was selected. The distribution of USMES 
cind comparison classes in the Student Study, by school and grade level, - 
is shown in Table 2. 

As shown in the "table, the psacticdl limitations of availability of 
classes resulted in (1) differences -in the grade-level distributions from 
school to school and (^TVithin some. schools a mild imbalance of grade 
levels between the USMES and compari^n grpups • . Ftirthermor^ , our evidence 
about the comparability of USMES and compcirison schools ist limited to the 
principals' ' judgments (and some students' achievement test scores). These 
restrictions .'limit our ability tb rtile out some challenges to ovfl& findings 
and to generalize from the findings. Nonetheless, we believe that the set 
of classes in the Student Study represented an informative diversity of 
USMES experiences and tl^ best . available soiarces of coii:5)arison data* 

The QARPS was administered to each of the classes recorded in Table 2* 
The Pencil Prob^lem-, however; was administered to only a sample of the 
students in each class, because the Pencil Problem could be adjninistered 
to ojaly tliree. students at a time. (The QARPS was administered to a whole 
class at a* time.) The selection ^of a Pencil l^roblem sample from each class 
went through five stages : 
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TABLE 1 . . 

CHAR^ftEKtS'TICS OF SCHOOLS 
[All figures are approximate) 





A 


' B 


C 


D 


Number of Students in 

* 

School 


300 


800 


700 


1,600 

* . ' '11*. 


Geographic Region 


Mid-central 


Southwest 


. North-centrai 

• < 


t 

1 

Southeast 


Community Population 

G 


50,000 


1.5 million 


25,000 


650,000 


Community Type 


Small, city 


* 

Large city 


Suburb 


Large city 


Surrounding Neighborhood 


Expensive houses 
and university 


Inexpensive ^ 
houses -and shop- 
ping center 


Moderate to ex- 
pensive houses 
and apartments 


Moderate houses 


Racial/Ethnic Composition 
of School' 


99% white 
1% A^an, Black 


49% Spanish sur- 

' named 
49% white - 
2% Black, Asian, 
Native Ameri- 
can 


99% white 
1% Black and 
Spanish sur- 
named . 


.98% Black 
2% white, 
Spanish sur- 
named, other 



mE2 

.DISTRIBUTION OF USMES 'AND COtgASISON CIASSES 'lN.,STUDENT STUDY ' • ' • ' • 



Grade 
Level 


SCHOOL A' 


SCHOOL B 


SCHOOL C ' 


• SCH 


OOLD 


3OTAL 


USMES 
Classes 


Comparison ; 
Classes 


liSMES 
Classes 


Con|arison 
Classes 


USMEs". 
Classes 


Coinparisor 
Classes 


.USMES 
Classes 


i 

CoDiparison 
Classes 


USMES 
Classes 


Cosiparison 
'Classes 


2 




2 
















2 . 


2-3 


2 




• 












2 




3 


h 

V 




z 


1 










2 


1 


3-4 


k 








2 


2 


.1 




3 


' 2 


4 






2 


'. 1 






1 


1 


3 ' 


2 


4-6 


4 •. 


1 














4 


1 


5 






1 


1 


1 




3 


1 


5 


4 


5-6 








1 






1 






* 

1 


6 






* 

1 








1 

' 3 


: 1 , 

.1 


S ' 


1 ■ ' 


TOTAL 


6' 


' 3. 


7 


4 


3 


4 


8 


f.; 

' 3 . . 


24 


14 
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!• S-tudents who had participated in the field-testing of 
the Pencil Problem in the fall of 1976 were excluded.* 

2 . the remaining students were divided into two sections : 
' those who had participated in USMES before the 1976-1977 

schobi year and those v*xo had hot. - (There wer^' students . 
with past USMES ea^rience in many comp2u:ison classes, as 
well as many USMES classes*) The schools provided infor- 
mation about past USMES experience fpr students in each 
class* of the sample. ^ ' 

3. If a class contained more than one grade level (e.g., a 
combination of second and third-grade students) / three 
students were chosen at reuidom (using a calculator program 
to generate r2mdom numbers) from each grade level, in each 
class section (past/np past USMES experience) • Each ^oup 
of three /-students so chosen wais administered the Pencil Problem 
as a group. If' there were less them three students available 
in any category (e.g., "third grade, no past USMES experience"), 
then that category -was dropped f6r that class; dropping of 
such categories was necessary to ensxire that the Pencil Prob- 
lem was always administered to homogeneous groups,' of three 
students . 

* r 

4. If a class . contained only one grade level, two groups of three 
- students each were chosen at random from each section of the 

class. Each group of three students was administered the ^ 
Pencil Problem together. There was, one exception to this-^ 
rule: . in USMES classes which appeaired to have been relatively 
inactive (i.e. whose teachers had reported doing less than 20 
hours of USMES this year), we selected only one group of three 
students from each section.** Again, if there were too few 
students available to foirm a complete group of .three students in 
any category, we avoided forming groups with leses than three 
students (e.g., if only five students were available in a 
^ category from- which we intended to form two groups only one 

. group was formed instead) . 



. *In the pretesting, only USMES classes (or classes expected to do!. USMES) 
Had s been involved. Sinco^the *pre-test experience could have significantly 
. affected students ' performance on the spring Pencil Problem, and since 
almost all of the affected students would have been in USMES classes (dis- 
regarding the few classes expected to do USMES which' actually didrno USMES 
this year) , to have included these students would have meant introducing 
a potentially serious bias into the spring data- Excluding these students 
reduced the available number of students in most USMES classes by up to 
six students. On the average, less than 15% of the students in USMES 
classes and less than 5% of the students in comparison classes were - 
excluded. The excluded students had been chosen by their teachers as 
"typical" of their classes. Their exclusion may have caused a mild reduc-" 
*tion.in the representativeness of the sample, from USMES . classes especially, 
but Jihis loss seemed less hazardous than the potential bias which would 
have! resulted from including them.. - f 

**Sinc^ a very few classes which appeared to have been reiai^ively^JJ^active 
were later found to have been more active th^tn^ it had appeared ,„-^hese 
^ classes are underrepresented in overall USMES results. J We vlo not believe 
cni/^" this problem seriously distorts our 'results since it oc^nirred.;^ infrequently. 
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5Ar> We selected r at ramdomr "standby** students within each cate- 
gory as replacements for students who might be absent. When 
possible , we selected 50% as wany standby students as the 
number of students originally selected in *a category • (This 
^ nianber proved sufficient in all cases.) ^ 

0 

wUhe final distribu'tion of Pencil Problem groups (each group con'taining 
three s-tudents) is shown in Table 3. Hot. surprisingly, the distribution 
shown in. this table resembles the distribution of classes shown in rable 2. 
Hence, ^:he constraints on generalizing from such a distribution which were 
discussed ahov& apply to Pen<;il Problem resxilts as well as QAKPS results. 
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TRB1E3 

DISTRIBUTION OF PENCIL PROBLEM GROUPS * 



Grade 
Levels 


SOBOC 


L A 


SCH 


OOL B 


SCHOOL C 


•SCHOOL D, • 


r 


COTAL 


\ 

USMES 
Classes 


Comparison 
Classes 


USMES 
Classes 


Comparison 
Classes 


. -I ■■ 
Classes 


pAttinariQon 

Classes 


USMES 
Classes 


Comoarison 
Classes 


USMES 
Classes 


Comparison 
Classes 


2 




8 
















8 


2-3 


5 
















5 


■ 


3 






h 


L , 










4 


2 


V4 


> 








5 


. 6 


3 




0 

8 


6 


.4 






4 


2 






2 


3. 


■ 6 


5 


4-6 


11 


5 


y 












9 11 


• 5 


5 






3 


3 


3 


6 • 




4 


5y, 13 


13 


5-6 . 








■I 












. 2 ... 


6 






6. 


^ 




— — 


■ 8 


4 




4 


TOTAL 


16 


13 


17 


; 9 


8 


r 12 


20 


11 


61 


45 



*Each group contains three students. 
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THE DSMES PROGRAM IN .TEE STUDENT STUDY SCHOOLS" " « 

Publications of the USMES project have described its progreun dn the 
foUcwing ways z ^ 

■ , • ■• •■ , " ■■ . ' » 

fnified Sciences and Mathematics for Elementaary Schools: 
thematics and the Natural,- Spcial, £ind Cononunications 
Sciences in Heal Problem Solving (USMES) was formed in xe^ 
sponse to the recoinmendations of the 1967 Cambridge Conference 
on the Correlation of Science and Mathematics in the Schools*' 
Since its inception in 1970, DSMES has been funded by the 
National Science Ibundation* to develop and carry out field 
trials of interdisciplinary xmits centered oh^long-range in- . 
yestigations of reaJL, ^nd practical problems (or "challenges") 
taken from the local school or contaunity environment."*. 

"Children working on USMES units tackle problems in getting to . 
^ school; problems ih the classroom , the lunchroom> and the 

playground; problems in consuming, producing/ and advertis.ing 
goods; problems in fiiiding the best way ^6«^^^arTX things , to ^ 
make decis|.oi^s, and to communicate certain kinds of information. 

. They 'work on real problems like a dangerous crosswalk near the 
Qphool or classroom :fumiture that doesn't fit the/ students in 

• the class .... 

- 7USMES engages students in all jispects of the" problem-solving 
process: definition of the problem; obsezsvation; measurement; 
collection of datai; analysis of data using graphs , chaxts, ■' ^ 

r statistics; discussion; formul-atien and tried, of solutions; 

develofanent and clcirification of values; decision making; work- ^ ^ 
ing with others -in small groups; and cccamanicating findings to * 
others. In addition, stud^ts become, more ^iiiqui sit 
critical in their thinking, cind more self-reliant. "** . 

,USMES classes in^ the four Student^Study schools used a wide Vciriety of 
USMES published xinits, and several locally-developed vmit^, as shown, in 
Table 4. ^ . ^ " ^ 

The total amoxaht of time devoted .to each USMES unit ragged from 4.5 
hours to over 100 hours. Most xinits took between, approximately 10 and 50 
&ours. Usually (but not always) aJLl the students in an USMES class were 
involved in USMES activities. * 
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*USMES Guide, EDC, 1976. 
**USMES brochure, EDC, 1976. 
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TABLE 4 

TJSMES UNITS USED IN STUpENT STUDY SCHQQLS " 



Published USMES Units 



Niainber of Classes 
Whi&h Used Unit / 



Advertising 

Bicycle Transportation 

ConsuHier Research 

Describing People 

Designing for Human Proportions 

Design Leib Design 

Dice Design 

Growing Plants 

Manufact\iring 

Mass Connnuhications s 

Orientation / ^ 

Planning a Special Occasion 

Pr^ecting Property. 

School^oo ^ 
to - • 

Using Free Time 

Weather Predictions /' ; ** ^ 




1 
1 
1 

2 

,1 . 
1 

\ 
5 

1- 

1 

3 

•2 

•1 

1 

,2 
\-2 ■ 



^ Locally developed Units 



Numbeijt of Classes 
Which Use^ Unit 



Keeping Restrooms ClBaiiJ 
Compiling a Cookbook 
Fire Sa^fety . ■' 

Iii5>rpving School 
Metric Awareness ' 
" ^^^^urcihasing a School Mat 
ISchbpl Spirit 
^Television, Sxir^ey 
Traffic in School 




1 
1 

.1 
3 

•1 
1 
1 
1 
1 
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DATA COLLEfCTION PRDGEDURES , . ' . , 



Three members of the USMES research staff visited each of thej four 
schools, in turn, dgring April an^ May 1977. They administered the 
QAKPS to each class in the sample, readir^g the items aloud if' students 
had any difficulty in reading the items for themselves.. The s^Doe' staff 
members administered the PenciJ. Problem to the selected groups of'Nthrefe^' 
students, using a procedure which ,ensu3:ed that administrators never knew,* 
at the time of a Pencil Problem adminls£ration , ' f rom which class the 
threev^students had' come » That pfocedtire wets to have , each administrator's 
next group of Pencil problem stud&its 6e selected (more or less at random 
out of all the remaining groups) and brought to the administration area 
by one of the other administrators. We took great pains' to ali^^e by this 
procedure, in order to prevent administrators from beingf biased in their 
observations or in their conversational tone'. - 
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DATA iiNALYSI^ METHODS 



Dctta Processing 



Data/ were entered into computer files and formatted with speciallyr 
developed computer programs idiich performed a number of verification pro- 
cedures to ensure accTiracy of the data entry. In additiort^ a number of 
entries. were spot-checked^ and basic cross-distributions were amalyzed 
• to further verify the accuracy of data entry. Scaling/ receding, and 
' statistical c Aculations' were performed with the computer programs of the 
Statistical Package for Social Sciences (SPSS) , Version 6* 



Coding of Outcome Variables 

. For purposjBs of cinalysis, students'- responses on the Pencil Problem 
cinfe QARPS were |:oded ^s follows: 

1. Pencil Problem r Total Number of Factbrs Considered. 
- • For this outcome variable, students received one point 

for 6Jteh (of nine possible) critical factors that they 
mentioned in discussing the quality and cost of pencils 
* and erasers. The possible factors were (1) how well a 
pencil writes, (2) how well a pencil can be sharpened, 
* (3) how well a pencil eiTases, (4) the cost of a . pencil, 
^ (5) the kind of lead in a pencil, (6) the size of a 
' j]fci)encil, (7> the strength of a pencil^- (^) how well an 
^ eraeer erases, and (9) the cost of an eraser. TheSfirst 
'seven factors (thoise concerning penqils) might have been 
mentioned -at any of three places in the Pencil Problem — 
in response to ."How will you decide ^on a pencil?" or 
"Which pencil do you think (your principal) should buy 
^ smd why?" or on the "Advice to Principal" sheet. However, 

\, students received no.in9re than one point for,-fach factor, 
no matter how many tidies they mentioned it. 



3. 



Pencil ^Problem: Number of Investigations Carried OUt* 
On thi^ variable , students received one point for each ^ 
pencil they wrote with, one point for each pencil they 
sharpened, one point *for each pencil- they erased with, 
cuid one point "for every other Jcind of ' investigation (like^ 
comparing the lengths of pencils) they carried out. 

pencil. Problem:* Response to "How Would:; You Convince Ot^er •. 
Students That the Pencil You Chose was the Best^^te?" ^ 
On this variable, students received one poirit for each pen- 
cil, factor they said they would "show other kids" or "tell 
other kids about." In addition, they received a point if 
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they said they would "let other kids try the pencils out^ 
{or the equivalent) ^d another point , if they said they 
would ask other students for their opinions about pencils. 

OAHPS: Attitu&es Toweurd Working on Real Problems and Producing' 
Effective Solutions*' All the QARPS vairiables, .including this^ one, 
involved summing students' responses, over several QAHPS items. 
The possible responses on each item had beexfc coded as. follows ^- 
(with a few exceptions noted below): \ 

• "Agree a lot": +2 ' ^ ^ 

• '* Agree a' little": +1 

"Disagree a little": -1 - . 

^ • ?Bisagree a iot" : -2 ^ 

The eight QARPS itemia^n which the Attittades Towsurd Working 
on Real Problems variable -was based included: , 

■ — • ' ,^ ♦ ^ ' ■■' ' . ' 

, - • "If kids show, that a^ change is. needed, then grownups 

will go cilong." ' 

• "I figure out what I 'Id going to' do before I start 

^ something." ^ - * . \ ^ 

^ . - ■ ■ ■ ■ " ■ 

m "Kids usually can't come jjp with good ways to solve 

problems . arotand school . " ( Coded in reverse /e.g., 
"Disagree a lot" coded +2.) - 

"Usuially there" is just one way to solve a real life 
problem." (Coded in reverse.) 

• "I like to make things better aroiind school." 

• "I think I am. good at solving feal- life problems." 

• "As soon as I think of one way to solve a problem, I* 
don't think about other ways.'" (Coded in reverse.) 

• "I^ like working on problelns*' that" have irfore than one 
answer." " « ^ ' / - 

QARPS: Attitudes Toweurd Group Interaction. v* * 

ISiis variable was based on the following seven QARf'S items: 

• "I like to wo^k in small -groups with other students." 

• "I like to help decide what we do in class." 

• "I like to work by myself." (Coded in reverse.) 

• "If I think other kids won't like my idea, then I ' 
keep quiet about it." .(toded in reverse.) 
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• "I like to ask questfions. ^' ' 

• like to talk to other students about xny ideas." 

^ • , '• . .'**'. 

like to listen to other students talk about their Z'' 
ideas. " . . '.^ ' ' " . " 

6,. QARPS: Attitudes Toward* Specific ProbleinTSolving Activities. 
\ This variable was based on the following five QARPS items: 

f "I like to write about xny ideas." " ^ 

^ • "I like to ineasiire things when it helps .solve real life : 

problems. ^ i . - . , 

• "I like to make charts and g-raphs that shcwr things I • 
■ found out . " 

• "I like to use nuntoers to ^olv^e rdal life problems." • 

• "t like to do surveys^ to find out what other peopl6 

, * - > . ■ think. . ; v ^ ■ . . ' , ^ 

> - ■ ■■ ^ ' ' ' 

While these six variibles-^^rere admittedly "crude -coxnposites of students/ 
responses, they allowed us to aha^ze a manageably small, relatively cp- . 
herent set of out<femes. Time did *not permit an analysis of the technical 
qualities of these ^composite variables, but we believed that they were 
logically related to the six outcomes on which the Stndl^t . Study^ was . focused. 



Stages of Analysis 



t 



^ Prior to the formal analysis of results, the Pencil Problem dai;a were 

"aggregated" into mean results for e.ach administration group. The adminis- 
tration groups were the groups of three -students to whom the Pencil Problems ' 
had been administered at once, and. aggregation had the effect^ of producing 
one s^t of average scores for each administration group rather 'thcin one set 
of scores for each individual student- 

■ 

The aggregation, was necessary to meet a teahnical reqnirenfent of the 
statistical significance tests used iii our analysis.' The significance tests 
we used* assume: that eacji observation, on which the test is based* is "indepen- - 
idbrit" of other observations. However it . was c^lear -to' adr^nistrators of ; . . 
the Pencil Pro]3lem that the three students in 'e^ch- administration group ^ere 
often ?influehced,Sy the responses of, the, other students "in their group • 
There seem^ to be no basis for treating individual stiidents ' Pencil Problem 
resp6nses as^ independent observations, xahder the/ circumstances. To have ^--^ 
done so would have -inflated the number of supposedly ind^fiiendent conservations 
s-ince there were three^ times as many .individuals as grougp?^ The inflated 
number of observations would have produced artificially f^5^^|^able significance 
levels, since the significance tests are. "strongly affected by the number - 
of observations. Aggregating the Pencil Problem data into administration 
group mean scores prevented this problem.- ' 



*These were primaurily F-tests fbr^ analyses of varian^ce and for multiple 
regresrsions. 
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For similar reasons, we experimented with a different kind of aggrega- 
tion of both QARPS and Pencil Problem data. For these experiments, we 
divided each class into two sections: (1) students with USMES experience . 
before this year andf (2) students with no USMES experience before this year. 
Then we aggregated the data into mean scores for each class section. Our 
purpose was to examine^ the 'possibility that the' scores of studWts with 
conpairable USMES experience ^ both this year and in the past, should be treated 
as '•not independent." In these experiments the • aggregated class section data 
produced * results similar to those produced by the origilial data. ^ Therefore, 
we have not included class section results in this report, except for two ^ 
samples in Appendices G and H. The coinplete" results for class sections, 
are available from the USMES office at Education Development Center^ 

■ ^ ' ■ . ■ ' ' \ 

Our main analysis of the Student Study -data took place in two stages: 

(1) examination of each outcome variable's apparent relationships with^ 
each independent variable**, considering just ohe independent variable at 
a time; and (2) analysis of each -outcome variable's relationships with each 
Independent, variable** after all the other i||ependent variables were taken . 
*into effect. The second -stage of analysis 'iMe allowances for incidental 
relatlbnships among the independent variable^ like a disproportionate 
number of non-USMES students coming from, certain grade levels. 



■*We also' conducted a separate siet of analyses 'using a modified coding of . 
past' USMES experience. ,This coding only applied to students '"who Had 
indicated (on the cover of the QAKPS) a different amount .of past USMES 
experience from the amount given in school records. Ultimately, we 
opted for using the school records in cases of conflict, to be consistent 
with our sagiplljig strategy.- However, results of this modified coding 
are also available from the USMES office. 

**Achievement d^a were used only ,in the second stage o^ analysis and only ^ 
•for ^Pencil- problem outcomes. "In these analyses, coinposite achievement 
levels (confining mathematics and verbal scores) , expressed in months 

\above or bj^low' grade level:, (adjusted for the date of test administration), 
were 'used. As noted above,- such data must be interpreted cautipusly. . 



RESULTS 



In the first stage of ^fed^sisr >e examined the mean scores of each 
outcome vsuriable by amount of USMES (see Table 5).* ^ 



TABLE 5 




• > 


• \ 


MEAN SCORES ON EACH OUTCOME VARIABLE 


Bli^ AMOUNT OF 


USMES 












Outcome Variable 

/t ^ ^ 




Amount of 


USMES 


None 


Little 


Substantial 


PENCIL PROBLEM 

Total Number of Factors Considered 


3.01 


3.52 


> 

3.35 


Number of Investigations Carried Out 


6.64 


7.09 


6.90 


Responses to "How Woiild You Convince 
Other Students That the Pencil You 
Chose was the Best One?'* 


- 1.03 


1.19 


1.15 


QARPS 

Attitudes Toward Working on Real 
Problems and Producing Effective^ 
Solutions 

. ' / ■ . . ' 

Attitude^^^^;^wax 

Attitudes Toward Specific Problem- 
Solving Activities 


4.11 

* 

■ , -. 4.Q2X 
2.03 


/ . . 

^.7i-. 

4.41 " 
3.01 

\ 

V 


• 

.-'.5.14 

• ■ 

5.10 

2.44 



: Table 5 sliows that the mean scores for Substantial amount of USMES 
exceeded those for None on all Six outcomes. Surprisingly, the mean scores 
for Little USMES 'cire the highest of all on foxir outcomes and the lowest on one 

However^ mean scores alone do not show the whole pattern of results. ^Fqr 
inst£uicV/ some differences between.'^mean scored are so small as to be statisti- 
cally ^iiisignificahtv consider!^ of error involved. Table .6 shows 



*Appendices A-F give more detailed- results for • the QARPS. 
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the "one standcLrd error confidence intervals" around the xneem scores shown 
in Ted^le 5. A confidence interval may be interpreted as follows: allowing 
for chance errors in saanfeLing and me astir emejfit ^ we can feel confident that 
about two thirds' of thgiMjm , scorels wd 'might obtain in similar circums.tcuices 
would fall within ^^^j^^^Bk^^CiL. .Also, about 95% pf these mean scores would 
fall within an intervS^^^ls^jas large (a "two standard error" interval) . 



TABLE 6 . ^ /. .. 

ONE STANDARD ERROR' CONFIDENCE INTERVALS AROUND MEAN SCORES 




Amount of USMES 



Outcome Vaxiables 



None 



Little 

X 



Substcuitial 



PEa^CIL PROBLEM 

Total Number of Factors 
Considered ] 



2.81 to 3.21 



3:^25 to 3.79 



3-20 to 3.50 



Number of Investigations 
Carried Out 



^^.01 to 7.27 6.27 to 7.91 6.43 to 7.36 



Responses to "How Would" You 
Convince Other Students 
That the Pencil You Chose. 



0.85 to 1.21 1.05 to 1.33 



1.06 to 1.24 



was. the Best One?' 



Attitudes Toward Working on 

Real fxoblems and Producing -^^3.80 to 4.43 3.41 to 4.02 
Effective .Solutions i 



4.94 to 5..34 



Attitudes ^Tc^ard^ Group 
- Interaction \ ,^ 



3.72 to 4.31 ; 4.11 to 

* ■ 



4.71- ' 4/90 to 5.30 



J 



Attitudes Toward Specific 
Problem- Solving Activities 



1.74 to 2.32 2.73 to 3.28 2.26 to 2^62 



Table 6 indicates th^t on all six outcomes there is some "overlap" among 
the confidence intervals for None, Litt^.?, and Substianti al . Figures 1 tlirough, 
6 show these overlaps in gra^ical /fonm. / ^ 



ERLC 



Tabl-e .6 and- Figures 1 through 6|fshQi/ that the one standard error confi- 
dence interval for Substantial. Amoais* ox/'USMES act^cilly overlaps that for None 
on all the Pencil Problem outcomes afcd on Attitudes Towcird Specific Problem- 
Solving Activities. The conf idenc(B|intexval for Little overlaps either the 
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' Confidence Intervals and Means for None (N) , 
Little (L) , and Substantial (S) Amoixnts of USMES 



^00 
3.6)3 
3.00 



} 



N 



Figure 1. Total Num- 
ber of Factors Considered 
(Pencil Problem) . 



8.00 
F.50 
7.00 
6.50 
6.00 



N 



Figure 2. Number of 
Investigations Carried 
Out (Pencil Problem) . 



1.50 
1.00 



t 



+ +■ 



N 



Figure 3. Responses 
to "How Would You Con- 
vince Other Students 
That the Pencil You 
Chose was the Best One?" 
(Pencil . Problem) 



5.00 
4.50 
4.00 
3.50 



N 



Figure 4. Attitudes 
Toward Working on' 
Real Problem^ and 
Upoducing Effective - 
Solutions (CARPS) . 




ERLC 



5.00 
4.50 
4 . 00 
3.50 



t 



N 



Figure 5. Attitudes 
Toward Group Interaction 
(QARPS) . . ■ 
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3.1)0 
2.50 
2/00 
1.50 



t- 



t . 



N 



Figure ' 6 . Attitudes 
Toward Specific Problem- 
^olv^g Activities 
(QARPS). ■ 



Itone or the Substantial interval on all outcomes except Attitudes Toward 
Specific Problem-Sol^^ng Activities.* - ^ 

Table 7 shows that there were some pronounced differences among the 
results for the four different schools. It is interesting to note that each 
school ranked -first on at leasff one outcome; school D ranked first on 
three outcomes. 



TABLE 7 . * 

MEAN SCORES ON EACH OUTCOME VARIABLE BY 'SCHOOL 



School 



Outcome Variable 



B 



PENCIL PROBLEM 



Total Nxomber of Factors Considered ^3.40(2) 2.94(4) 3.37(3) 3.45(1) 



Number of ' Investigations Ccirried 7.36<2) 5.59(4) 7.85(1) 6.85(3) 
Out - ' ' ^ 



Responses to "Ho^.would You 
Convince Other Students that 
the Pencil You.Cho^e was the 
Best One?" 



1.00(3) 1.49(1) 1.00(3) 1.02(2) 



OARPS jf 



Attitudes Toward/Forking on 
R^al Problems and Producing 
Effective Solutions, 



6.16(1) 4.47(2) 3.22(4) 4.42(3) 



r 



Attitudes Toward Group 
Interaction 



4.87(2) 4.85(3) /2. 95(4). 5.46(1) 



Attitudes Toward Specific 
Problem-Solving Activities 



1.87(3) 2:67(2) 0.74(4) 3.69(1) 



Note. Numbers in parentheses show relative' ranks of schools. / 



*An analysis of variance confirmed what the^se figures suggest: that the 
' superiority of Substantial over None isftonly statistically significant at 
the .05 level on two outcomeis: Attitudfes Toward Working on Real Problems 
and Producing Effective Solutions, and Attitudes Toward Group Intetactiort. 
Similarly; the dif f-erences between Little Amoiint of USMES and ^the othe.r two 
AmoTints of USMES are only statistically significant on the outcome Attitudes 
Toward specific Probleii- Solving Activities. (At the .33 significance 
level, both the Substantial and the Little group als^scored significantly 
better on another outcome—Total Number of Factoi!^ CoSidered. ) 
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Table 8 indicates that there were also noteworthy differences by grade. 
However, the pattern of results by grade is more consistent than the pattern 
by school: the upper grades produced the highest mean scores on all outcomes, 
and the sixth grade yanked number one on five of the six outcomes. 



TABLE 8 ■ - 

MEAN SCORES ON EACH OUTCOME VARIABLE BY GRADE 



Grade 



Outcome Vairiable 



PENCajL 'PROBLEM " 

Total Niimber of Factors 
Considered 



3.13(3) 2.95(4) 3.51(2) 3.13(3) 3l.60(l) 



Number of Investigations • 
Caurried Out . 



5.67(4) 4.88(5) 6.46(3) .7.66(2) 7.95(1) 



Responses to "How Would You 
Convince Other Students 
That the Pencil You Chose, 
was the Best One?" / 



1.07(4) 1.05(5) 1.22(1) '1.09(3) 1.16(2) 



QARPS 

Attitudes Toward Working on ' 

Real Problems and Producing , 5.96(2) 3.40(5) 4.20(3) 4.19(4) 6.11(1) 
Effective Solutions ^ * ' * 



.... — 

Attitudes Toward Group 
Interaction 



4.00(4) 3..20(5) 4.33(3) 4.69(2) 6.37(1) 



Attitudes Toweird- specific 
Problem-Sqlvincf Act^ities 



2.15C4) 2.22(3) 2.56(2) 2.06(5)^ 3.18(1) 



Note. Numbers in parentheses show relative ranks of grade levels. 



The pattern of res^ults by school and by grade suggests a ne^ to treat 
the earlier results by ^ount of USMES with some caution. As su^ested by 
Table 2 (page 12), a disj^roportionate number of the classes in School D and 
in the sixth grade were USMES Classes. ' Therefore^ the apparently favorable 
conditions in School D and in^'the sixth grade may be somevaat confused with 



USMES effects. 



ur 
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A possible way to resjfl.ve this confusion is to excimine the Amoxint of 
USMES results school-by-school and grade-by-grade . TcdDlefe 9 and 10 show the 
mean scores on each outcome variable by Amount of USMES within each school 
arid grade , respectively. ^ 



TABLED 

MEAN SCORES ON EACH OUTCOME VARIABLE BY SCHOOL AND AMOUNT OF USMES 



^ ^ Outcome Varicd>le 


School 




y - ' 

C Amount of 


USMES 


* 


t' ' 








None 


^ Little 


Substanrtial 


PENCIL PROBLEM 


. A 


3.21 


3.61 


3.42. 




'■■ 


2.75 


2 .67 


3.12 ' 


Total Number of Factors 


* C 


3.50 


3 . 72 


.-=■'3.00 


Considered , 


D 


2.67 


3.76 


■ . • 3.65 




' A 




• 6.44 


7.31 


Number of Investigations 


B ^ . 


6.2i' 


3175 


5.76 


Carried Out 


C ' ' 


■ 6.83 


9.78 


7.17 




D 


5.29 


7.24 


7.33 


Responses to "How Would You 


A 


0.83 


1.44 


0.91 


- Convince Other Stud€^t§ That 


B 


1.58. 


1.00 


a. 57 


the^ pencil You Chose was the 


C 


1.11 


1.00. 


0.92 


Best One?" 


D 


. 0.57 


. 1.24 • 


. 1.12 


QARPS * 

Attitudes Toward Working on Real 
Problems and Producing Effec- 
tive Solutio'ns 


A^ 

B ■■ 
C 


■ 6.78 
3. 64 
2.82 ■ 

. 4.76, 


' 5.08 
z - y / 
3.^)3 
3.54\ 


6.41 

■ - 3.50 
' -4.93 . 


V . > 

». 


A 


4.52 


4.90 


,4.92 


Attitudes Toward Group 


' B 


■ 4.14" 


3.60 


[5.-51 ■ 


Interaction 


^ C 


1.94 


2.58 


3.59 


- ' D. 


5.32 


5.01 


- 4------ 


Attitudes Toward Specific . 


■ A' 
B 


'0.41 

£2.03 


3-17 
3.07 


1^71 
2<97- 


' Problem- Solving 'Actxyiti*es 


c . 


'o.39 


0-97 


0.83 


^—^ — " 


D 


. 4.(^6 . 


" 3 . 59 


3.62 













i 
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TABLE 10 

HEAN SCORES ON EACH OPTCCffg: VARIABLE BY GRADE AND AMOUNT OF USMES 



Outcome VariedDle 


Grade 




' Amount of 


USMES ^ 


None 


- Lxttie 


Substiuitial 


PENCIL PROBLEM 


2 


2.87 . 


3.56 


3.17 




3 


2.42 


2.93 


3.40 


Total Number of Factors ^ 


4 


3.43 


3.94 


3.36 


^ Considered 


5- 


,3-l'3 


3.4^4 


3.04 




6 


2.67 


3 . 78 


3.76 




2 


5.87 


• 5.33 


5.67 




3 


3.67 ' 


5.40 


5.33 


Number of Investigations Carried 


4 ' 


"7.14 


8.06^ 


5.36' 


out 


5 . 


8.50 


8.78 


6.86 




6 . 


4.56 


6.33 


8.96 




2 


1100 


- I -'44 


0.67 


Responses to "How Would You Con- 


3 


'1.08 


• .i';4o. 


jO.67 


vince Other Students; That 


4 


lao 


6,89 


1.44 


the Pencil You Chose v^s the- 


- 5 


1.20 


1 . 22 


0.98 


Best OiEie?'' 


6 


0.33 


l.ll 


1.33 


QARPS 


2 


7.00 


4^ 


. 7.25 




3 


3.33 


2.36- 


3.86 


Attitudes Toward Working on Real * 


4 


3.60 


3.84" 


4. '65 ■; 


Piroblemfi and Producincr Ef fisc~ 


s 


4j03 


3.59 


4.54 


tive Solutions' 


6 


5. 33 


3.85 


6.8P 




2 • 


4.00 


4.21 ^ 


r - 3.25 " 




3 


2.55 


3.18 


■ 3.61 


Attitudes Toward Group Interaction 


4 


4.32 


4.16. 


4.39. 




5 


4.23 


4.31 


5.06 




6 


5.33 - 


,5.82 


6.68- • . 


\ ■ 












2 


-0,50 


3.79 


2.13 




3 


1.49 


3. 45 


2.17 


Attitudes Toward Specific Problem- • 


4 


2 .,97 


. 2. '60 


• 2.32 


Solving Activities 


5 


1.86 


2,34 


,2. or 




■- 6 •> 


2.44 


3.77 


- 3 . 16 



In general. School B shows below-average benefits from USMES on problem- f 
solving outcomes but above-average benefits on attitude outcomes. School D 
shows the opposite pattern of results: Grade 6 shows above-average benefits on 
cQl outcomes, especially problem-solving outcomes. However, the patterns shown 
in Tables 9 cuid 10 are , not uniform' because of the r^atively small number x>f 
sinidents in each entry. Furthermore, theses results "are hard to summarize be- 
cause therev are so many of them. . 
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A different approach to ' separating the effects of Amount of USMES from 
those of school and gtade is regression anjalysis. Tables 11 through 14 show 
. a simnnary of regression results for each outcome valriable and independent 
variable* 



TABLE 11 • 

REGRESSION A NZU^YSISppSUI.TS FOR AMOUNT OF USMES 

■ ,.' . . ■ 



■ Incr. 
Outcome Vairiables B* f " S.^. Other nif 

-■ ' " - - ■ " : ' ' . ' (B)* (%) (%) 



PENClIi PROBLEM/ 



T^tal Number of Factors Considered {'0.07 
Maeber^ of Investigations Carried Out \o.6i 
Responses to "How Would Yoii Con- 
vince. . . , 0.07 

QARPS . ' ^ 

■ / . ■ 

Attitudes Toward Working %n Real 

Problems. . . ' * 0.28 

Attitudes Toward Group. Interaction 0.33 

Attitude^* Toward Specific. .Activities 0^23 



' ' f ^ ' — 

*NOte. Column headings are defined in text below 



TABLE 12 ^ 
KEQRESSION .ANALYSIS RESULTS FOR SCHOOL 



0.16 


0.2 


11-4 


.66 


0.43 


1.8 


26.9 


- .15 




0.4 


8.9 


.." .54 


0.18 


0.2 


6.2 


.13 


0.18 


0.3 


6.8- . 


.06 


0.17 


0.1" , 


6.1 


, .17 




. r 







B* S..E. 


Incr- 


r2 ^ 


Sig- 


Outcome Variables (B) * 


r2 
^(%) 


Other 
(%) 


nif .* 


PENCIL . PROBI5EM " 








Total iivaJ^er of Factors Considered - , 


4.8, 


6.9 


■ .21 


Number of investigations Carried Out - - 


5.4' 


23.3 ^ 


.13 


Responses to "How Would You Con- 








vince ..." - - ■ - 


8-4 . ■' 


0; 9 


.08 


2ARPS . ' - . • y 
Attitudes^.Toward W9rking .on Real Prob- 














lems ■ , - ■ - ' 


■ 4.5: 

. 2.7 ;^ 


1.9 


-00 


Attitudes Toward Group Interaction ^ - 


4.4 


.00 


Attitudes Toward Specific . .Activities ^ - - 


5.9 


.0-3 , 


.00 



*Note . Column headings are defined^ in*' text below. 
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TftBLB 13 

REGRESSION /jaiALYSIS RESULTS FOjl GRADE 



1 * 



Oixtcome< Variables ^ 


B* • 


S.E. 

. (B) * 


1 • 

iVicr. 
(%) 


other 
(%) 


Sig- 
nif .* 


PENCIL Problem : • 
















Tbtal/NUinber of Factors Considered / 


0, 


.04 ^ 




13^ 


0.1 


11.6 ^ ' 




NrimMf o^ Investigations Carried Out 


1 


.18 




36 


9.7 




.00 


Re^K>nses.to "How Woxild You Con-- 
















■• viixce..A.' .. \- ' " 


0, 


.00 


0- 


0,^ 


6.0 


>-^^ 3 ; / 


.99 


















flftjC^Psudes Toward Working on Real 
















Problems. ... 


0. 


.62 


0- 


Ir4 


1V9 


4.6 - 


- . 00 ; 


Attitudes Toward Group Iritejraction 


0, 


.70- 


. 0- 


13 


^ 2f.5 


4.7 


.00 


Attitudes Toward iSpecific. .Activities 


-0, 


.12 


0. 


13 . 


6.1 


6.2 


.66 












. y 





*Note' . Column headings are definedf in .text below. 
TABLE 14 

REGRESSION ANALYSIS :RESULTS BY . ACHIEVEMENT , LEVEL 



4f 









• Incr.- 


•'r2 ■ 


sxg- 


Outcome Variables . 


B* 


S.E. 


r2 . 


' Other 


nif.* 






(B)* 


(%) ° 


(%) 




PENCIL PROBLEM ' . ^ \ ' 












TotcJ. Nxamber of Factors Considered ; 


• 0.04 


6.02 


5.4 ■ 


: 6.3 - 


. 03- ; 


Number of Investigations Carrie^ Out!^ 


0.12 


0.04 


6.5;.. 


;22;.l. 


> -01 


Responses - to "How W6uld You jCon- ' 










vince." ;. . v , . . 


^O.OO^ 


o.or 


•"■ 0.1 


■.':9.1 


.72 








— rf-T- 







* Note . Column headings are defined .in text below. - \ - _ » 

T . Achievement data were onlV available :fof.- the Pencil Problem .sample.' 

. : "In Tables 11 through 14<the coTumn headed "B" gives "unstandardized 
regressioij coefficients'V t^nventionally called "B's** in statistics^ A 
B indicates the expected 'difference 'in an outcome variable for each difference 
of one unit in the corresponding independent? variable. For instance, the B 
of 0.04 on the top line of numbers in Table 13 means that the expected^dif- ; 
ference in Total dumber of Factors Considered corresponding' to a difference 
of one grade level, .in this sample, is +0.04 Factors. 

For Amoimt of jUSMES, the 'units are quantities /of vUSMES experience .over 
one or more school 'years. - These units were .coded . as follows : j 



EKLC 



4j2 : 



0 f or no USMBS eaqperience, past or present; . 

- ■ ■ ' . ' ' ' ' - ■ • ■ ^ " ■■■■ ^ 

1 for "little" USlffiS experience — either some, amoixnt less, them 20. hours 

this year or else past DSlfes iexperience only? * ^ « / . 

2 for r substantial" US>&S experience — either e3(i>^rience in the past as 
■ well as this year or else over 20 hours of experience this year* 



Thus> for exainple, the B of 0.07 on the top line ritainbers in Table 11 means 
that students with "little" USMES es^erience woxild be e3?)ected to consider 
0.07 more factors in the Pencil Problem than students with no USMES experience, 
and^ likewise that students with "substantial" .USMES eacperience vrauld^^^^^^^ 
pected to consider 0*07 nore factors than students with, "little" USMES 
experi^ce. i ^ ■ . - ■ . • '\ 

For the indepe)ident variable Aciiievemen't Level ^ the "units are. months, 
above or ^i^w grajie lev^l in con5>osite mathematics and* verbal achievements 
Thus, the B "of '0.04 on the top line of numbers in Table 14 meains t±Lat the ex- 
pected difference in Total -Nximber of Facters "Con^iiered cprresponding to one " 
month of grade-equi^^alent^" achievement is 0*.04* However, the figureis for 
Achievement Ibevel should be xead with speciaT caution^ as indicated on page? ■ 
9 and 22 . - "'^ . ' ■ " . . • . \ .: 

.' ■ ^ ' • ' ^' . \ ' ^ • ■ . , 

Thea^e are no entries in th^ column headed ' "B" in Table 12, becaoise there . 
are no "units ^ to express the different categories of the indep^dent variable ;^ 
School. B's would not *be!appropriate statistics in Table 12. ^ 

The .column headed . "S.E; tB) " in Tables 11 through 14 gives "Standard > 
Errors" of the corresponding Bjs^ These 'Standard errors are 'analogous to thei.; 
:'stand2trd errors- of onsda scores that were used to create Table 6^ The standard :• 
erro^^pf the B's may. be interpreted 'as. follows: allowing* for chance errors;., 
we cl^feel confident that approximately 68% of the ;B^^ 

circumstances would JEall within , ope steuidard error :of /.the\r^ B'^s.- and 

that /approximately 95% of similar!? comguted B*s wotild fall; within two - standard 
errors of .'the* repoifted B's. Thus, for example, about 68% of the /siiii^-airly 
computed B's for Total Nuiiibe'r of Factors Considered and" Amo^pt of* USMES (top 
line of Table 11) should fall withi-n plus or minus 0.16 (one standard' error) . 
of 0.07 (the reported B) , and about. 95% should fall within- 0.32 (two standard 
errors) of. 0.07. ' . . ^ ' \ : 

^ The coltamn headed "R^^ Other (%)" '^and the coltuan headed yincr. R^" in 
Tables 11 through 14. involve "squared multiple correlation coefficients" or 
R^"^s. ' These are stat5.stical measures-of the relationship between an Outcome 
variable and one or more independent variables. "Incr.\.^l^" , or' the increment 
in R?, may be interpreted as the percent of the total variation in an, outcome 
variable which is "acpounted for," or explained, by:^- corresponding indep^n- ; 
dent vfiriable after the effects of all the other independent' variables -are - 
taken into /account . It gives a conservative estimate of the. effect of . an 
independent variable because it , removes from consideration not only the effects 
.of Other *5jidepeihdent 'variables but also any joint effects of two or more . 
'independent variables. For instance, the joint effects of Amoiimt of USMES. af^d 
of Grade (which are impossible to separate because, th^re were a disproportionate 
number of USMES classes in the, Grade 6 sample 1 axe removed from , consideration 
when the increment in is computed for either Amount, of USMES, or for Grade; 



. < . ■■ • • ■ 5?- . 

Hence ^ the incrMient in provides a relatively con^rvative measoire for 
both Aipount of USMES and Grade, ^ C ■ ■' ' 

V The nuntbers ±n the coiumA headed *'R^* Other" represent the ^fects of 
*alf other independent ^vauri^les (and any inseparable joint effects) for each - 
increinent in r2. The sum of an increment in R^ and its corresponding "R 
Other" represents the percent of an outcome variable to taa ^riation which , 
is accounted for by all of the independent variables, "•^ . .. 

. • . . ' / ' ■ • ■ ■ ■ ' ' ■ . • 

As an exan^le of the R^ measures, consider the top line of T^le 14, 
-showing results, for the outcome. variable Total Number of Factors 
and for the independent varialile Achievement Levels. In this case^ the";J^Rr ' 
Other of 6. 3% means- that all the independent variables except Achievement 
Level (i.e. ^ Amount of USMES / School, and Grade > accounted for 6. of the 
.variation in Total Number of Factors Considered, The increment in R^ of 5 •4% 
means that Achievement Level accounted for 5.4% of the variation in Total 
Number of'^Factors Considered after all tiie other independent variables were 
accounted for. , In all, 11.7% (6.3%^+ 5.4%) of the variation in Total, Number 
of Factors 'Considered was accoxmted for by all the in^ependezxt variables. The 
remainina^8.3% of the -variation was not accounted for; presu mab ly it was due 
.to chance *ef fec$:s and to unmeasured factors. 

While the incremient in R^' can Ise useful . for con5)iu:ing the influence of ^ 
different independent variables, its practical significance depends on the 
situatjlon. . Fpr- example, it may be interesting to no^fe^^^iJiat School accounts 
for more of the' vsLriatioif in Attitudes Towar<i Working on Real Problems and - ■ 
Producing Effective Solutions .1±tan does eittie^- Grade or. Aaroimt' * 
however, for/teache'rs and" administrators,' School an3 (Srade may be unchangeable^ 
7givens. Spioxint of USIjES' may the only variable ^which represents an : ' 
: opportunity ' for improT>ing these; attitudes. ' In tois c^ 

: increment " in/R^ may not be as important as the iact- tha*t the- increme^ — and ^ 
-the :B~are- simply ppsitive'^^-i^r Amount of , -USMES;* V - " 

Hh^ column headed 'l^^ignif." in Tables 11 through. L4 .gives the si^ifij 
ca?^cey4evels for corresponding incremerfts in R^' and^ standard errors, of B 
.QJie .significance level applies to both) . The significance level represents 
the probability that the* reported results could. have occurred by chance alonfe, 
and a low significance 'lej^el^ is therefore desirable • jHow low .a significance 
level, "should be" •depends! on the risks which an educator may be ready tb ^ 
accept. For scientific research, it is common to look for significance levels 
of .05 or ' lower, i.^. ? ncTmore than a i-in^20 chance 'that a ^reported "finding" 
waS:^ really due to luck. / However, a school principal who is willing to accept 
a risk of up. to one ch^mce in three, say, because lie of" she believes that the 
potential benefits o^weigh the risks, should presumably be willing to accept 
a significance levea ,of up to .33. ^ ;. - 

.' A few general Aobservations can be' made about .the results reported in. • 
Ta]^les ,11 through iVl First, , it is iit^ortant to remember that the lower 
significance levels on QARPS outcomes are pa,rtly a result of the much greater 
sample size for QARPS am^lj^s (about 1000 individual students) than for 



*Note that Achievement Level is one of the . independent variables for the' Pencil 
Problem putcomes^ but not for the /pARPS outcomes 1 ■ - ^ 



Pencil Problem/ Analyses (cibput lOQ administration groups of three students 
each); leurge 'sample sizes tend to give low significance levels. 7^ in^rtant 
pattern appears throughout the "Incr. R^" and "Signif * " columns^ of Table 11: 
each of the i3idependen£ variables- has strong relationships with only some 
. of the dependent variables. Amount of USMES has its strongest relationship^ 
-with Number of Investigations Carried Out (in terms of increment in R^) and 
with Attitudes .Toward' Group Interaction {±n terms o'f significance level); 'it 
has modest relationships with .the other two Q'ARPS outcomes and almost- 
negligible relationships with the other Pencil Problem outcomes. School has 
very low significance levels 'on the QAKPS outcomes but more modest significance 
levels on the Pencil Problem- outcomes (this may be in part an artifact of the 
large QARPS sample -sizes, since S'chool's increments in R^ are only a" little 
larger fdr Pencil Problem outcomes thcin for QARPS outcomes). Grade has rela-- 
tively strong* relationships- with Number of Investigations Carried Out, Atti- 
-tudes Toward Working on Real Problems and Producing Effective Solutions > and 
Attitudes Toward Group Interaction, but 'practically negligible relationships 
with the other outcomes. Achievement Level has\ relatively strong relationships 
with Total Number of Factors Considered and with Number^ of ' Investigations 
Carried Out (despite the fairly large amounts o^ missing data on achievement) 
but almost no relationship with responses to "How Would You Convince Other 
Students That jthe Pencil You Chose was the Best- One?" 

In reviewing Tables 11 through 14 it -may" |)e appropriate to recaU. that, 
for reasons discussed above, the regression analysis may somewhat understate, 
the possibly complex effects of dif f eren^^^amounts of USMES experience. Any 
non~lihear. effects ^e^-g. , situations where Little USMES produced positive 
results of. greater magnitude th.an Substantial USMES) would riot improve the 
increment jLn r2 or the significance le vie 1 for; Amount of USMES. Furthermore r 
some effects of USMES may haVe -been obsciired by .the imbalance, in the sample 
of classes (e.g., in Grade 6 and School d) . Therefore, it may be, best to ^ 
balance the impressions gained from Tcibles 11 through 14 with those gained 
from the earlier tables when drawing conclusions about the effects of USMES. 

Finally, the results for the QARPS questions directed to USMES students 
are' shown in Tcible 15. . \ ' . . , ' 

The-f«BSults of Table 15 parallel those of several -similar, earlier studies 
(see, f oir^'exampile r page« 38 of The USMES Guide , '4th ed.')^ Efeth the overall . . 
very positive attitude toward doing ^USMES- work and the divided opinion about 
whether USMES is "h^d work" have been observed in all the earlier studies, al-. 
though the exact format of . the-^uestions — as^well as the times and locations 
of the studies — have varied- V 



TABLE 15 
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K: 



TO 


STODENTS WITH USMES EXPERIENCE 




■ ^* 








^ * 
















• 


RespQjQse 


- , ■ 






"Agree 
a lot" 


"Agree 
a little" 


"Di'sagree 
a little" 


"Disagree • 
a lot" 


21. 


' r think USMES Work is fun. 


74.2 • 


19.0 


3.1 


• 3.7 




x tinxnx .Uo.ni:«o woztjc xs xx^xxn^. 








69.2 


23. 


Doing iJSMES is hard wprk. 


16.0^ 


34.2 


22.0 


27 .8 


. . . 

24. 


I don't know why we do some 
things in USMES. 


16 5 ' 

X V » 'J 


28.5 


26.0 


29.1 


25. 


Doing USMES makes me think. 


59.4 


28.6 


7.4 


4.6 , 


26^ 


I think USMES^work is con- 
fusing. 


9.8 


20.1 


26.7 


43.4 


27. 


I think -USMES work is . • 
important* 

. » • ' . ■ ^ ■ 


68.2 


21.0 


7.3 . 


" 3.6 


28. 

4 


In USMES it ' s hard . tg 
decide what to do next . 


22.0 . 




24.7 , . 


16.9 . 


29. 


i would like to do more 
USMES. 


70.2 " 


16l7 


6.4 


6.7 
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: CONCLOSIONS . 

Effects of USMES on Students " 

' "Ehe mean scores of the Substantial Jtoount of DSMES group were higher . 
than those of the None group on every outcome, and the mean scores of the - 
Little Amount of USMES group were higher than those of, the None' group ,on 
every outcome but one*. However the Substantial Amount of USMES group did 
not always , score higher - than the Little Amount of USMES group. (See Table 
5, p. • 23, and Figures 1-6, p. 25.) Also, these effects were only , statisti- 
cal!^ significant on some of the outcomes (see footnote on page<26) . 

The magnitude of these. USMES effects were generally small , compctred to 
the effects of Schooi, Grade, and Achievement Level. On each outcome th^Js^' 
.was^t least one other independent variable with a markedly larger influence 
than Amoiant of OSMES. Also/ the effects of Little Usmes were often erratic: 
on four outcomes- the Little Amoxint of USMES' group had a, mean score above 
both the, Siabstantial and the None group^ while on einother outcome the Little 
AmQunt of USMES group had a mean score below both the otiier groups. Partly 
for these reasons, a line^ regres.sion analysis showed only moderate signi- 
ficance levels (.06 to .17) on four, outcomes for Amount of USMES and poor 
significsmce levels (over .50) on the other . two outcomes. One 'Of the > ' 
statistically -significant regression coeffecients was negative (see Table 

11, p.3o)V- ^ ' ' ■ ^ : 

, The effects of Amount of USMES were, most pronounced in a few settings, 
such as School D. The None group in School D scored the lowest of all four 
None groups on all the Pencil Problem outcomes, yet the Siibs-cantial Amount 
of USMES group in School D scored the highest of ali four Subs tcintial -groups 
on two of these outcomes and second"— highest on the third (see Table 7, 
p. 26) . Similar effects of Amount of USMES were found in the 6th grade (see 
Table 8, p. 27) 7 however, since many of the 6th graders in the sample came 
fropi School D, the 6th grade and School D results, may be related. 

The motivational effects of USMES did not seem to be so localized. The 
responses to questions directed at students with USMES experience confirmed 
once again a repeated finding: the 'great majority of USMES students enjoy - 
USMES work, find it important; and would like to dp more (see Tcible 15, p. . 
35). This finding, appeared to be true -in all the schools and at all- grade 
levels (see Appendices D and E, pp. 41-42). ^ ^ ' 

Factors Influencing the Ef fectiveness 'of USMES f 

Why weren't the motivational" effects of. USMES translated into more 
consistent xesults pn this main Pencil Problem and QARPS putcomejfe? We ccin 
only speculate on this issue, but it appears that several * factors were 
important. K 

Little amounts of USMES (USMES experience in the past only, or. less than 
20 hours of USMES. in the current year with no past experience) evidently 
produced inconsistent results. Although earlier research (se<e Chapter V of 
The USMES Guide , 4th ed.) has shown that time spent^ on USMES ,is unlikely to ; 
impede "basic" skill$ development, neither is a brief "escposure to USMES 
certain to •improve problem-solving skills significantly. Substantial amounts 
of USMES experience — including amounts greater than the 20 hours which defined 
"substantial" in this study — may be nece^ssary for consistent, significant 
results. 



V 



*More6ver, of the 20 QARPS items completed by ali students, there were five 
items that produced statistically significant results for Amount of USMES; 
on all five of these items, the Substantial Amounl?^ of USMES group producoa 
the most favorable responses, and on fpur of the five the Xittl^e Amount of • 

-USMES group produced the next-most-favorable responses, (see Appendix A, p. 38) 
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The specific activities done in an USMES class may also be crucial. 

•The USMES school Study*. found evidetice that opportunities for skill develop- 
ment were often closely associated \^th the occurrenc§; of serious investiga- 
tions of data ^ extended discussion^ Stoong students, and other recommended 

<^BM£^ activ^Lties. Fiirthemiore . in the present study the Substantial Amount 
cJf ufeMES group appeared to have engaged in significantly higher levels of 
many recommended activities than the Little Amount of USMES group (see 
Appendfs^pV P- 43) ^ and the Sxibstantial group produced the most consistently 
positi^^.^result^* « - 

InSkddLtion, the surprisingly large effects of School (which included 
some coii*&nity differences in this study). Grade, and Achievement -Level 
suggest^j^at all of these factors make import^t contributions to the impact 

« of USMES^ experience. Certainly they should not be overlooked in future 
research on USMES and related programs. 

Finally, the influence of the school administration sJioul'd not be ig- 
nored. In School USMES had its most positive effect on attitudes , axid 
in School D, USMES had its most positive effect on problem-solving skili,' 
In each school, these were the outcomes that the' j)rincipal of the school 
desired most to promote (see the USMES School Study report). The particular 
style of USMES ^hosen in each school emphasized the desired outcomes and Wc 
successful in improving it.. . , 

Some Implications " ' , , . ' t: 

. - The results of this study suggest that the choice of USMES units (and 
the .local development of units) should be based . on a careful consideration 
of the outcomes that would be most valuable for a particular school and class; 
Furthermore, teachers should be aware tha^i|idividual students may experience 
a wide range of effects of USMES experiencdi. While virtually all students 
seem to enjoy USMES work, there seems to b/ considerable variation from stu- 
••dent to student in just what is learned.** For consistent, significant results,, 
this study and the USPlES School Study suggest that "regular" USMES -units ^ 
of some duration—with recommended USMES teaching methods —should generally 
be preferred over abbreviated or haphazardly-created units. . . [ 

In reviewing the results of this sjiudy, it seems clear that additional 
research is needed- on the impact of extended (especially; multi-year) ^ USMES 
^d on real .problem solving at upper grade levels.. The trends in this -study 
suggest that these varieties of USMES may prove especially fertile f or -the 
*developmen-t of effective ptoblem-soLving abilities and attitude^, ^^..gesearch ^ 
on a local or regional level may prove just as valuable as national r^earch 
on these issues. - ^. 

'.For diagnostic and other kinds of research, we believe *that the QARPS . _ 
and the Pencil Problem may be useful instruments. The QARPS .is easily adminis-: 
teifed, easily analyzed, and shows satisfactory discrimination. The Pencil 
Problem probes real-problem-solving. abilities more deeply than "pencil, and 
paper** tests yet can be administered and analyzed in practical ways. 

Adda^^fonal discussions of the findings reported here are needed; We 
'hope. that readers will join us in those discussions. 



: *Available^ fr®m the UJSMES project office at Education Development Center. 

**A comparison of ApPendix • H" (p. 45) with Table 11 (p. 30) shows that the 
' average effect"^ of USMES on groups • of students may be more predictable 
t3ian effects on individual stu'dents, ^ * 
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APPENDIX A - • / ^ 

Percjentage^of Students^espondlng "Agree a lot" or "Agree a little" on QAKPS 
Items 1-20, by. Amount og USMES 

/ 







j 

QARPS I±em/ 


Amount of 




oignixi— 






None 


Little 


^ Substantial 


cance* 




1. 


If kids shqiw that a change ^ 
is needed,/ then growaups 
will go along. 




67.1 


• 

62 . 8 








I like to/ work in small 
groups, with other students. 


84.5 


86.5 


§^4 






3. 


I ll^e to write about my 
.'ideas. /. - 




70.3 


62.0 


B 




"5" 


I figure out what I^m going 
to do before I start some- 
thing./. 


_88_^8__ 


88^7__ 


88.5 


._ — . — . : — ._ 




. 5. 


Kids usually can't come up 
with good ways to solve 
problemiS: around school, - 


53.4 




39.6 


A 




"67 


I like to help decide what 
we do in class. . 


82.3 


87.8 


88.6 






7. 


Usually there is just one 
way to solve a 'real life 
problem. 


58.6 


58.6 


48.2 


A 




8. 


X like to measure things 
when it help3 solve real 
life problems.^ 


67.7 


75.0. 


73.4 ^ 






9. 


I like to work by myself. 


~6073. 


""5975 ■ 


""5677 






10. 


I like to make things 
better -around school , , 


92.2 


89.6 


90. 9 






11.: 


If I think other kids won't 
like my idea, then I keep 
quiet about it. 


76.2 


70.7 " 


58.3 . 


_A_^, 


12 . 
13.' 

1 


I like to_ask 'questions. * 

I like to make charts and 
graphs->Jthat show things I 
found out i 


__69_^1__ 
64.5 


__73_^9__ 
64.7 


73_.4 

60.0 






K ' 


I think I am good at 
solving real life problems* 


54.1 


56. 


__^__62_^5__ 







15. 


I like to talk to other 
students about my ideas. 


77.3 


77.7 


80.5 






16. 


I like to use numbers to 
solve real life problems. 


: 47.8 


56.1 


51.4 






17. 


As soon as- I think of one - i 
way to solve a problem, I ^' 
don't think abqut other ways. 






.39.1 


A 




. 


students talk about their \ 
ideas. 


89.2 


86.0 


V ■■■ 

. 90.4 . 






19. 


I like to do surveys to' 
find out what other people 
think. • 


78.0 


76.9 


82.7 






20. 


I like working on problems 
that have more than one 
ajiswer. ' s 


74.0 


70.0 


74. S 





*Key. A: /£ < .01; B: 



Oli.2 ^-05; blank: 

49 



05 < £ (chi-squared test) 
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APPENDIX B 

Percentage of Students Responding "Agree a lot" or "Agre e a little" on QARPS 
Items 1-20, by School ^. 



QARPS Item 



School 



B 



Signifi- 
cance* 



If kids show that a change is 
needed, then grownups will go 
along^ 



■ v 



I like to work in small groups 
with other students.V 



^56^7 
86-0 



1: 
A. 



6- 

"8^ 



"l like to write about iriy( ideas . 54^9^ 

figure out what I'm going to 
do_bef ore I_start something. 87 > 4 

"Kids usTially can^t come up 
with good ways to solve prob- 
lems around school^. - 

"~like~to help^decide'what we 
do in class. 



68.0 51.6 


74.8 


A 


82.2 78.8 


'89.5 ' 


A 


66.5" 49.2 


74.5 




.90.7 , 86.4 - 


88.9 • 





Usually there is just one way 
to solve a real life problem^ 



24.8 


50.0 


34.8- 


62. 8 


89.3 


86.5 


85.3 


. 86.7 


32.6 


58.4 


51.6 


62.7 



9. 
10. 

ii' 



12. 
13' 



I like to meaaure^ things when 
it he]jps solv^real life prob- 

lems. ; 

I like to_work_b2_^self^__ 

'~I~Tike~to make things better 

around school^ . 90. 2 

~If I think other kids won't . ~ 
like my idea, then I keep 

guiet about lt^_ = [ 60.7 

~I liketo ask questions. 



75.7 



6i:o 



55.0 



65-0 



7 7 J. 
*56.0- 



92^5 ,83^6 94^3_ 

65.5 70.1 66.2 



75.7 



70.5 



-67.2 



75.3 



14. 
15' 
16^ 

IT! 

18" 
20" 



"l^lilke to^malce charts and graphs 

that show_things_I_f ound out-. 53.7 

I think! am good at solving 

real life problems. 59. 6 

'i like~to talk to other" 

47..0 



stu- 
dents _about_m2_ideas^_^ 

T~like to~use nimbers to solve 

real life problems. 

~As soon as I think of one way 
to solve a problem, I don't 

_H^^i5iL.§^2HH_2£^}£H_^^^--- 

I like to listen to other stu- 
dents talk about their ideas. 

'~l"^like~to do surveys to find 
out what other_people_thinkj;^_ 

'~I~like 'Sorking'on'problems 
that have, more; than one answer. 



66_^1 46^7_;_ :Z3^i. 

54.8 _51^6_: 67_^0_ 

79_^3 ^^8^1 85_^4_ 

53.7 * 45.1 -56.6 



_A_ 
_A_ 
_A^ 

B 



30^8_ 

85_^1_ 
75. 3 



43^_ 

.11^1. 
76.8 



Alll. 
_81_^4_ 

_78,1_ 

59.3 



58_^9_ 
_90^8_ 
_85^5_ 

76.8- 



_A_ 

A 



*Key . 



2 < .01; B: .01<._£<:. 05; blank: •05^<£ (chi-squared test) 



50 



o 

ERIC 
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APPENDIX C • . , . 

Percentage of Students Responding. "Agree a lot" or "Agree a little" on QARPS 
Items 1-20, by Grade 



QAR?S Item 


1^ 




Grade 






Slgnlfi- 


2 


-3 


4' 


5 


'6 


cfflice* 


1. If kids -show that a change 














is needed, then grownups 
will go along. ' 


70.9 


56.1 


68_^5_ 


62.7 


68.5 


B 


2. I like to work in small 

groups with othei; students. 


80.0 


79.9 


80.7 


87.0 


90.5 


- A 


I likito^^write about my 
idea< 


54.5 


65.9 


64.3 


58.9 


68.3 






3S.2 



87.3 



70^ 
'67.3" 



92.7 



I fljgure out. what I'm going 
to d|o before I start some' 

-^thing^_^. ; 

"57"Kids~sually can't come up 
with good ways to solve 

problems around_school^ 

"67"l liice^to help decide what 

we do in/class^ 

7. Usually there is just one 
way to solve a real life ^ 

_ EE25i£2j:— ^^-^ 

"8. I like to measure things 

when it helps solve real 

life problems. 

"97""l like_to_work_b2_52self ^ 

107""B7iike to make things 

better around school_^ 

11. If . I^think'^other kids won' t 
like my idea, then I keep 

^ 3^i^^_§^2y^_iii-_ « Z^i.§ 

12. >^l]]like to ask quegt ions ^ 

13. * I like to make charts and 

graphs that show things I 

. . f ound out^_] .\ ^ ^ 

1A7. I^ think I atn^good at solving 

real life problems. ' 

157~7like~o"'talk to other 

_ students_about_m2_ideas^_^^ . 

16-."l like .to^Use nufiibers to 

solve ££§l_lif£_££2^i£5Si._ 

177~As"soon as I think of one. 
way to solve a problem, 1 

187"^l"ike to listen to -other 

students talk about their 

^deas^ ^ 

19- I like to do^'surveys to 

find out what other people 

^jthink^ ^ j_ 

20. I like working on p'roblems 

that have more than one 

answer.' 



'92.1 86.7 87,0 89.7 



86.6 



4|7_^2 44.8 42^5 



T 



_84^3___87^0 89^_ 

41.8 . 64 . 6 57 . S . 53.0 41.3. i^A^ 



72^4 



86. a 



70^3 
'63.1" 



73^0 
■5^.5' 



74.8 
■47.3" 



91.9 90.6 93.7 



.81.1 
'73.2" 



68_^4 64^9 49_^8 A 

;72_.9 ^70^1 75_^2^ Lll-I-Zi 



52.7 


61.6 


64.0 


59.9 


65.2 




49.1__ 


_54^9__ 


_53^8_J 


_62^0__ 


66.4 




72.2 ' 


76.2 


77.8 


78.0 


85.6 




56.4 


50.0 


51.2^~ 


49.7 


54.1 





_54_^3_ 
_86_^0_ 
72.6 



48.0 50.2 31.8 



98.2- 



86.2 87.7 94.1 A 

_ _ N;^ 



83.6 



78.2 75.0 



79^8___78^7_,__88_^3 
72.8 *69.6 76.2 



*Key . A: £ < iSa; B: .01<£< .05; .blank: .05<£ (chi-squared test) 
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Percentage of USMES Students Responding "A eree a lot" or "Agree a\ little" 
on QARPS Items 21-29 » by School ' 



. . . SchooL ■ Signifi- 

QARPS Item b" C D cance* 



21. I think USMES work is fun. 96.8-90.9 91.0 97.2 / B 



i2. I think USMES work is boring. 15.2 , 11.0 >14.0 15.0 

^ ■ 



3. Dojjig USMES is hard work. 53.0 42.3 46.5 ^ 55.1 



2<). I don* t* know why we do some . » . 

' • • 52.4 36.8 48-.5 \44.0 B 



things in USMES. 



25. Doing USMES makes me think. 



I^hink. 86.1 8»;3 84.8 90.2 



26. I think USMES «ork is confusing. 33.7 31.5 22.0' 29.6 

■9 ■ . ' : 



27. I feh: 

•»0 ' T_ TTi 



ink USMES work is important. 



.84.7 • 95.1 75.8 93.1 - . a"^ 



28.'In USMES it's- hard to' flecide '. . ^, - ' „ V 

what to- do next. . ; 61:r 54.0 54.9- 60.9 



29.- I woiild lik«to do more USMES. 79.9 87.7 '89^0 90.2 



Jim 



*Yjs7. A: .01; B: .01<_£<.. 05; blank: , . 05 < p. (chi-sqiiared. test) 
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APPENDIX E ' 

Percentage of USMES Students Responding "Agree a lot" or "Agree' a little" 
on QARPS Items 21-29, by Grade . . ' ' . 



J 



QARPS Item 



Grade 



Signifi- 
cance*- 



21. I think USMES work is fun. 100.0 86.1 91.6 93.2 _ 97.4 



22. I think USMES work is 
boring. 



0.6 » 16.8 17-.4 17.9 



6^8 



A 



23. Doing USMES is hard work. 
24. 



^ 



il don't know why we cio 
Some things in USMES. 



50.0 , 49.5 49.4 ..47.1 •*-54.2 
45.0 59.4 51.9 45>-l^ 31^9 • . Ar* ^ 



25- Dbing USMES makes tae think. 85.7 86.1^*87.6 85.9 91.5 
26: 



I think USMES wbork is 
confusing. \ 



40.0 41.2 3U4 32.5 



19^ 



27. 



28. 



I think USMES work is 
important. 



In USMES it ? s hard to 
decide what to do next. 



A 



94.7 94.1 89.7 82.0 92.6 



68.4' 70.3 61.9 56.6 . 49.7 



29. I would like tcx do more 
^ USMES. 



90.0 82.2 



88.9 89.5 



A: £ <- .01; ,B,t .01<£< .05; blank: .05<£ (dhi-squared test) 
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APPENDIZ F 



Percentage; of USMES Stiidents Checking Each Response on QABPS Item 30, 
by Aao''wyt.of USMES (Llttrle vs.' Substantial) 



Besponse 



Amount of -USMES 



Little 



_ Signlfi- 
SubstansClal, cance* 



30. When we '.do USMES, I usually do these 
^. things: ^ 

measure things . 



39.6 



35.8 



do survej^s <5f peojple 



buil4" things 



32.0 
35.1 



47.1 



41.6 



ask, questions 



42.-2 



writhe about things we f|.nd' ouf 
work in small groups 



^.0 



41.3 



S2.2 



>60.0 



72.4 



A 
A 



make . graphs or charts 
do math 



help decide what' we do 
talk about: my ideas ' 



f work by ^^^elf 



20.4 
25.3 
47 . 6. 
'42.7 
.23.1 




.63.3 
"22.2 



»Key . A: £. < . 01 ; B : . 01< £ < . 05 ; blank: . 05 £ . (chi-squared test) 
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Sample of Class -Section Resalts: Mean Scores on Each QARPS Outcome by^ 
Amount of USMES* * ^ 



i, • ■ '■ * . . 




Amount of 


USMES 


Outceme Variable 


None 


Little 


Substantial 


QARPS . -^ '^ ' , - - 

Attitudes Toward Working on Real Problems 
and Producing Effective Solutions 


4.45 


2.91 


5.49 


~v • : ~ 








Attitudes .Toward Group Interaction 


3.92 


3.87 . 


"5.15 










- . . . - b ^ 
Attitudes Toward Specific Problem-Solving 

Activities / 


1.52 

- — ■ J 


2.58 


V 

2.49 



*Comparat)le . to bottom half of Table 5.. "C^^^ 
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APPE^IX H 
Sample 



of USMES* , 


■ ■ \ 




\ 




Outcome Variable B** 


■ ■ * 

S.Es 
(B)** 


^ Incr. 
(%)** 


Other 
(%)** 


nlf .** 


QARPS . ^ • 


^ ■ 








Attitudes Toward Worjcing on Real 
Problems and Producing Effective 
Soliitions *• > -SS 


.33 


• 


21.5. 


.08 


Attitudes Toward Group Interaction .44 


* DKJ 


1 Q 
i • " 




• ±^ 

•» ■ . 


„ ... - : - . 










Attitudes Toward Specjtfic Problem- 

^olvfncr Art-LvitlGS ' .09 


,32 ■ 


0.9 


11.5 


.63 


*Compa"»'nbxe tc bottom half of Table 11. 










n**Columa headings are ^defined in text after 

1; . . ■ . 


Tables 11-lA. 




■■' 


- ■ ■ ■ ■ ■ ' ^ ■ ^ ' 




■ . > 










1 
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More Reseeirch on USMES and Basic Skills is Needed. 

_ ■ ' : ■ ' ^ ■ ■ .' 

We eelieve there were technical flaws of a very serious nature in both 
the 1973-74 data and the 1974-75 data used -by the -Boston University evaluation. ' 
team*. The USKES staff conducted a careful^ ^computerized einalysis of the ' 
original data^ during the fall of 1975 and the spring of 1976. . Among the ' • 
flaws we found were the following:. 

1. In the data collected during school year 1973-74: 

. Eight of the 12-^ studied classes contained misssing or erro^neous data^ v 
for ►which the investigators made no reported adjustment: . 

-■ • . ■ ' ' 'I " , ^ ■ . ^ . •- 

. Eleven of the 12 classeis^^d. invalid or incorrect student-identificatxon 

codes for at least some of th^ data, while seven of th^ classies actually 

Had invalid qr incorrect codes for the majority of their data; again, no 

adjustment for these flaws was rep^Qfted. . ^ . . .. 

2. In the data collected dxiring school year 1974-75: 

. Approximately one-half of the .data cards used in computing the reported . 
results had either invcilld identification codes or invalid (lata (e.g., 
letters where there should have been numbers ^ numbers in the wrong 
- = areas, etCH ) f- apparently the d^ta cards were automatically generated 

/rom machine-readable scoring Jsheets, without an. 'adequate screening for 
V*. : '^^i^y" '^ata {like stray -^pencil marks on.^the sheets , sheets, out. of 
• ' <i>rder, ^etcC) ;. \ ■■•'^ ■; 

• ' . .. Twerity:-eight -o^ the.' 60* classes contained |3ata thfi^t ^as biased :arid v ■ . 
/'invalid, because*' thfe ci.asses were nc^ .properly. matched;_- in fact, ^ere " 

were not even equiv^^lifent nvimb^ of observations at pfSuSti grade level— ~ 
. the ^SMES data contained more observations than the conrparison data . 
- at some grade levels emd less at . othier, grade levels- — even-^though the 
Scaling of the achievement scores ensured that scores would . vary with 
grade levels. ^ . _^ 

' ^ Because of these flaws 'in th^ data collected by the Boston University 
team, we believe that the impact of USMES experience on students' achievement 
scores has not' been adequately studie*&. : ^' 

In general, our own emalyses support the conclusions of :^e Boston Univer- 
sity team 'that, "Clearly, USMES students rdo not fall behind their cohtrol 
counterpaurfcs in their performance on tests of basic skills, " and that "though 
it is not -statistically significant (at the . OS level) , there is. a noteworthy^ 



*M.' Shann, An Evaluation of Unifjfed Sciences and Mathematics for Elementary 
* Schools Purging the 1973-7^ School Year , Bqston rUni varsity , Augx;st 191&7 and 
■ \. M. :Shann, N. Reali, H. Bendei^, T. Aiello, and L. Hench, Student Effects of an 
Interdisciplinary Gurxiculum for Real Problem Solving; The 1974-75 USMES 
Evaluation , Boston University, December 1975,. s ; 

■ ErIc • • 57 3 
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trend fpr the growth rates of the control classes to fall behiiid those 
for the trSMES classes at the upper. gradei leyefs.-"* However, we believe 
that more accurate r detailed evidence on these concliision^ is needed by 
teachers and administrators. . . 




i ■ . 



*Shann> et al- , The 1974-75 USMES I^^aluatiotl. 
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J/ APPENDIX J: 



\ 



Sample T^ache^r Weekly Sinmnary Form (including First ' 
Report on a Unit and Last Rejport on a Unit) and 
Teacher Interview Form 
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SUPPLEMENTARY FIBST REPORT ON A UNIT 








(Please enclose vith veekly summary for first veek. ) 
















How many students are in the class? ' ' ^ * 






2. 


How did the challenge arise? (Was tl\e .problem already being discussed by 








the students? Did you initiate discussion of the problem? Did the problem 




•I 




arise in some other way?) , * \ . ^ - 

* 


r 


■< ■ 












■ 






3.. 


How do you and the students describe the 'challenge^ (What is the ''wording" 


• 






of the challenge?) * ^ . . ' " 








f * ' ^ 
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TEACEBB'S WBBELY 




Tour name: 

Unit: 



Week covered by this report: 
School: 



Grade level: 



1. Please comment on student planning this veek, (Do. you think they 
much or too little time on planning?) 



2- Were any new groups formed this week? .(If su, please dei;:ribe the rroups, 
their tasks, and the process by which they were formed.) 



3. What obstacles did students encounter in this we^k*-r work?. 
How did they deal with them? 



. Did 'you conduct any skills siessions- t'h^ week*? . \ 
•c^^ What topics were covered? *• 

What made ycu decide that the session(s) were needed'? 



ERJC 
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5. Did you hAve to provide any redirection for the students this veek?- 
What sort of i»ntervention did you make? y 
What made you decide that intervention needed: ' 



6. Were there any activities you hoped dr- expected to arise, which did npt? 



?• What else do ve n^^^o know about vhat happened this week? 



8, We would like to have copies of any student work that you can send us. 



PLEASE BE SURE YOU HAVE OOMPLSniD riHE PDLLOWINS SCWgtfUES FOR AUL UMS SESSIONS THIS VEEK, ■ 



duration of session: 
MAJOR ACTIVITIES-. 



no. of students actively involved: 



•wliole class 
small group,. 



Briefly describe activities: 



DISCUSSIONS: 



_ whole class 
■ small group 



Briefly describe* discussion ( s ) : 



student-nin 
teacher-run 



at beginning session 
In middle of . Vession 
at end of session - " 



ERIC 
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duration of session: 
MAJOR ACTlhpTtfe: _ 



i^le class 
small group 



Tuesday 

_ no. of students actively involved: 



Briefly describe activities: 



DISCUSSIONS: 



vhole class 
smfiTI group 



BriefLy. describe cliscussionCs ) : 



student-run 
teacher-run 



at beginning of s elision 
in middle of session 
at end of .session 



Wednesday 



duration of session: 
MAJOR ACTIVITIES: 



vhole group 
small group 



no. of students iactively involved: 



Briefly describe activities: 



DISCUSSIONS: 



vhole class 
small group 



student-run 
teacher-rtin- 



at beginning of session 
in middle of session 
at 'end of session 



Briefly describe discussion(s) i 



: -63 
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r 



r 



Ttiursday 



duration of session: 
MAJOR ACTIVITIES: 



whole, class 
small group 



Briefly describe activities: 



DISCUSSIONS': 



vhtole class 
small group 



Briefly describe discussion (s ) : 



no. of students actively involved: 



student-run 
teacher-run 



at beginning of session 
in middle of session 
at end of sessipn 



r 



Friday 



duration of session: 
MAJOR ACTIVITIES: 



whole class 
small group 



Briefly describe activities: 



no. of students actively involved: 



DISCUSSIONS: 



whole class 
small group 



Briefly describe discussion( s ) : 



student-run 
teacher-run 



at beginning of session 
in middle of session 
at end of. session 



J 
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SUPPLEMENTARY LAST REPORT ON A UNIT 



(Please enfclose with weekly summary for last veek, ) 



r. Please comment on the :i5nding of unit activities* 
Was the problem finally resolved or -set aside? 
Tf it was resolved, what was the resolution?' 



^1 



2. What made the class decide that: work on the unit should end? 



3^ Please comment on hov effectively students were able to work together over 
the course of the unit- 



\ 



X 



# 
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USMES STUDENT STUDY 

^ . : - 

Teacher Interview 

NAME OF TEACHER [ S CHOOL, 

USMES UNIT • ' ^ _CITY_£_ 

INTERVIEWER ' ^DATE__ 



1.. DATES OF UNIT ' ^NO. SESSIONS PER WEEK 

AVERAGE IJSNGTH OF TIME PE^ SESSION : ^ . 

2. NUMBER OF CHILDREN ACTIVELY INVOLVED PER SESSION r 



3. CHALLENGE afi understood by teacher^d students: ( "wording" of challenge?) 
4-. Briefly descri'be the activities that took place: * 



5. Was the problem finally resolved, or set aside? 
What was resolution: 



OR _ 

What^made class decide 
tcr-S!et it aside; 




6, Were any small groups formed during work on the unit? 
How many children per group:' 

AND 

Tasks* of each grpup: 



66 



-56-. 



?• Did the mawar activities occur when the class was together as a whole or 
in 'Smair gnoups? 

(. * . 

8. Did moat discussions occur when class was together as a whole or iji small 
groups? ^ • . ^ 



Sr action teacher-run: 

AND t * . ^ 

.Fraction student^run: ' * ' * 



9* Did you conduct any sJcill sessions during this xinit? 

Topics covered: ' r 

AND ' . • ■ 

Why sessions were needed; • \ 

10. Did you have to provide any redirection for students at any point 
Kirt^ of intervention: 

AND " ' 

Why intervention needed:,* 



11» Please comment on student' tolanning throughout the unit. Do »you tlfink* they" 
spent too miich or too* littre^ime on planning? - - . 



12. Please comment on how effectively students Were able to work together over 
€he course of this unit. (What problems and 'successes did'they encounter?) 



s 
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The Pendl Probliem: An Instrument 

to Mrasure Real Pro^rii-Solving AUHies 
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\.. • \ 'J.<''-^' •>-*. . :„iiv. •■■,c.■ 

. "U:^IFIEP:. SCIENCES AliDr*»MATHE;iATK:-,S FOR ELEiiEHTil^ ^ 



!!athe|natics -and 'the Natural > ^oc'l'^l^ik and .CoBmunica±io: 



3i' 



> in'»'Real Problem Solving; 





This report was pxCTpred with the support of the 
National ^Science ^ow^ation , ^Si-ant No« SED-690107i. 
Any opinions, fi^^^s*, conclusions r or recommendsi- 
tions expressed ii^^ois publication axe those of the 
authors do not necessarily reflect the views of 
tlie National Science Foundation. ' '■ ■ ' ■ r 
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^IVELOPMENT OF, T|JE P^NCIIi PROBLEM : 



BACKGROUND 



.t 



Siiice no commercially av^lable * ijistrUments would adequately 

mefiisure real problem^solving skills ,had e^ver beeij located and since 
past , efforts^ to ^develop such measures had met with considerable diffi- 
culties, a major goal of the 1976-1977 USMES. re search /program .was to 
try to* develop, such asi instrument. 'An iittpox^ant requireioent for the 
i^nstrument- was -that it* also be suitabl? for dxagnostic us>e by teachers. 

• In 1971 Professor Bernard Shapiro' from Boston qniversity began 
developing instrxaments to measure . student prbblem-soiyoiig abilities. . 
The first such instrument/ used in 1971-1972> was the Notebook l>rob- 
lem. In this test individual student g are present^ed with three 'dif- ; 
ferent types of notebooks. Each student is asKfed to -select one note- 
book as best for his or her class and to give reasons -^f or the*^ selec- 
tion. Cost information's given to the students; paper^ pencils^ and 
rulers are availalDle^if- students, wish to use them. No time limit is 
imposed^ but the testing has always been completed in one class period 
Two outcomes are meawsvired^: , ^ 

■ - ■; ■ .5 . _ ' , ' . . 

• . Does the student name; any reasons for his pr her choice 
th?it could be objectiyeiy measured, ii^^ size^ cost/ or 
weight of the 'notebooks? (This variable^ is called 
. . \ /^Reasons for Selection. ") ' 




Are any of 'thW student^ s reasons based on actual tests? 
If not, are any basedl on suggested tests? ^ (This varia-^ 
ble-is csilled ''Highest Level of Warrant. ") . 

VcLTiables focus on .two" specific problem-solving skills— - 
ing quantifiable differehc^ among alternatives-solutions 
Igning tests to' measure those differences.-. 



In .1971-1972, samples of two to six students selected at random ' 
from^a selection of thirty-one -USMES and 'twenty-two non-USMES classes 
(matches for^ grade level and school) 'were tested with the Notebook 
Problem.- Dr. Shapiro's report* on this testing concludes, thaj: 



•in 



♦Bernard* J. Shapiro, The Notebod]C\Pj^blem: . Repoj^^ obsejnr^tlpn 
Pxohlem-SQlvlng Activity in USMES dnd Control Classrooms. EDC, 197. 



'terms *of -the two dependent variables stuai'ep/ the uiSMES epcperience 'liaclr 
irrespective of units or teachers inV?ftf^d , . a marked and positive ef^ 
^ feet on the. sttidents' problemr-sdlving behayitor." * OJhere^ were^cknow^ 
le&ged shortccmijigs i?i;^the design r admi nistration , and analysis, of l^s 
stu4y/ but cis noted iii: tiie report , the, "re^eini n^ aispect to these dif-^ ^ 
ficulties lies in the clarity auid .consistency o^^ actual results* " 

(For a suTumary^of the results, see Chapter V of - ; 

eiaition/ EDC 1976.):' . l' ; . • • , ■ 

. ;■. * ...... ;■' ■ ^ . . \ . ■ • " • 

, As part of a study ^fef economic learning conducted in 1975"197i$ / 
Arthur Ellis and Allen cn.enn of the University . of "MiJineso^ 
a modified Notebook Erol^tem (using written responses) to sevetr^-eight ' 
fifth and sixth gradi4rs. ^%ie students Were, assigned ait randOT 
four classes rec^viijg foiiir different' types of ,ixistruction (USMES Manu-- 
f acturing unit;* e'concmic games , arid sijQaulations ; economic workbook ; and . 
geography — as, a control) . .Uie USMES students .scored' highest on both 
Notebook Problem* outcoraes. Although the. nvanber of classes involved 
was limited, the results were again encouraging.'*^' ' ! ' " 

For the school year 1972-1973, .Dr; Shapiro and his assistant 
Susai> Rogers developed a new instrument called the Playground Problem; ' 
Groups of five students are selected at random from/sample classes and ' 
are asked to plan a new playground for their school. They are. provid- 
ed with a catalogue of playground equipment , cost data, measuring' in- 
struments, and a hypothetical budget of $2>000. The scoring protocol 
yields twelve group scores: four "behavioral" scores' (motiydtipn to . 
accept problem, commitment to task, efficiency in allocating jobs, 
and nature of group leadership) , four cognitive scores (variable iden- 
.tification, ^measurement , calculation and recording) r and four scores 
on the playground plan produced' (on the scale , labels, landmarks, . and 
area designations) > ; ^ ' '-p - 

For the 1974-1975 evaluatipii, -Professor Wary .Sh and her staff 
develop€id a third instrument in "this genre; th^' Picnic Problem. Groups 
of five: stiidents, selected at random from samj^le' classes develop plans 
for a class picnic. The studentis aire supplied/with a photograph show- . 
ing various foods . they may select and' a map showing the location of a 
school and three parks f ram' which they iaay choose' a fpicnic site. Mea- 
suring instruments, cost data, and a hypothetical 'budget of twenty- 
five dollars are also provided, 7 The scoring protQcol includes the : 
four behavioral scores and four cqgnitiv§ scores derived for the Play- ' 
grouild Problem. ' > , . T ' ' ■ ' ^ 

■ ■ - _ 

The Playground Problem wais admihis^j^jred in 1972-1973, 1973-1974, 
and ^74-1975. The Picnic Problem .was"^ administered in. -1974-1975. Un- 
fortdhately,^ Dr. ^ Shapiro has neve2?> reported on the 1972-1973 results 
anH the information gained^ from the later testing is limited. -Analysis 
carried out on the 1973-1974 . Playground "Problem data -yielded no signi- 
ficcint difference amongj^. the groups of classes for any of the outcome. 



— ■ i I - " ■ , 

♦Arthur K. . Ellis , and Allen D. Glenn , ■ Effects of Real and Contri ved 
Problem Solving on Economic Learni/agr .(in preparation) . ^ ^ .V 
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variables: in the view of the Bos)£oh University evaluat or s./ the best 
explanations for these results ,^p£obably "lie with the unsatis^factory 
testing procediires and with the instr^iment its eli."* 

. The. analysis of/khe I&74-1975 Playgroiand Problem and the Picnic 

• Problem, data alsp yielded.no significant differences between the ^ 
tJSMES and hon-DSMES groups. The report on these analyses again ques- 

' tions the" appropriateness ,of the ^instruments. The e valuators con- 
' cixided . "the roare objective instruments to mecisure. probjLem solving 
skills eure still too unsophisticated to give an acci^JOi^e reading."** 

The Playground and Picnic Problems. presented students .with larger> 
. more complex pr6blems than the Notebook Problem and on esi thought to 
be of more interest to. istudents^p' in, these ways ^ they reseirdiled -iaie 
real" problems of USMES units more closely than. the simple Notebook 
Prbblem:.. However ^ they were npt^-acamini^tered as real problems while - - 
the Notebook Problem^was; it b^oUld be >^ore^ solved in;, the - 

small cimount of testing time available and Jch'e problem and its vari- ' 
ables were more tangible and less\,hypothetical., T^ scbring pjSrptocol 
for the later problems .w^ also more ambitious anfi more' removed from 

the data itself . - * . ^ 

" . ' . ■ ' ■ ■ • '■ . ■ . * - 

For these reasons^ our best alternative for developingpfan' appro- 
■ priate instrument seemed to be continued development of an "Assessment 
of- Problem-rSdlving Approaches" (or APSA) , /which -had evolved from the 
Notebook -Problem. The APS A, as originally created in the fall 'of 1975, 
presented students with a~^more carefully scripted notebook selection 
prptlem. When revised" for field-testing in the fall" of l^JS, the. 
-APsk was still basically a consumer; research* probleta% .but s 
were asked this time- to" decide 'which of several, pencils they /thpught. : 

. the.ir school should "buy. for students to uise. Sttidents / e,sp^cially ;^ 

^ younger ones, were found to be more interested in penciiks. and' pencils . 

, pre^nted more options for choice.. Students were presented with sam- ^ i 
pies of the pencils and encouraged to "look over" the pencils and. to . v. 
test, them in whatever. ways they saw fit. They were told that the 
schpol would buy $10.00 worth ot pencils, and they were shown the 
price 'of ^each pencil. (To maintain the realism , of the. pr^oblem situa- r 
tion, after the testing was completed we did supply each ischool with > 

V $10.00 worth of pencils selected pn the basis of the students choices; ) 
students were also asked a number of questions about how- they decided 

• on. a pencil/ how they would convince . others that their, choice was the ^ 
best/^and related issues. The^r respQnses were tape -^recorded/ and the 
administra^;pr (a member of the USMES , research staff) took notes on 
their investigations. Typically^ the testing sessions took about 
fifteen minutes. , ' ^ . 



; *M,H. Shann, An Evaluation "of Unified Science and Mathematics for Ele~ 

mentary Schools (USMES) During the 1973-74 School yeaf. Bosjton Uni- 
versity, 1975. ' ' ' 

**M.H. Shann, N.C. Reali, H. Bender, T. Aiello, L. Hench , ,Stud^^ Ef-_ 
fects of an Interdisciplinary Curriculum for Real Problem Solving: 
- The 1974-75 USMES Evaluation. Boston University, 1975.'' : 

ER?c • . ■ • :'72^ ;■: ''''■''^■^{'y : 



. V The fall fi^ld-±es^ting: involved. 125 studente 
of tie 1976-1977 USMES Student Study. .The students eame from classes, 
grades l-6r>^ebedxjled^ to do USMES this''^ school 'year . They were ' select^ 
fli by teachers. a& "representative" of stiodents" in.- 

. .Whilis ther^vas a c<^^ we deliberately, altered .a 

ew parts of it during the field-testing\to eiperiinent with^.-I^ 
iinprovements . We recognized that these alterations would produce data^ 
jwliich were^ndt ^fully standardized / /cut the . experiments made important 
caatributions , to the development cff the instr^al^t, laiey shbwed, fat 
exan5>le/ students were l^ss reticent when tested three at a, time* 

This strategy th>2s produced more iM&rmatioh thaui testing students in- 
dividually, while* it .^Isro allowed for testing a larger. sa2i5>.le of stu- . 
'dents. Purthe'rmore; we found that it was easy for the administrator 
to note. differences in students' strategieiS/ such as the! nuiaber " of 
iE^^cils they wrote with or the -reasonis tiey gave for .tiie^^ 
even when several*^ students were working on the APSjC^t once> .'Student 
xesponses. also suggested a number of refinements in the wording of ■ 
.'questions and in the materials which the students saw and used during . 
APSA sessions. ^. . 



Students' responses to the APSA questions were transcribed* and 
studied primarily to suggest further- revisions in the instrument. ' 
Overall, however, we found that students were .eager to work on- the^ 
problem of choosing pencils. They/also expressed confidence in their, 
ability to make good recommendations about pencils., and they were able 
to make their choices within a reasonable amount of time. . (Most .stu- 
dents took 3-5 minutes.) ' V . . ^ ^ . 
^ ■ ..' ... . - ' ' " , " ' ' ■ '■' 

i- I THE PENCIL PROBLEM: PRESENT VERSION 



The spring version of the instrument, which has come to be known 
as the Pencil Problem, consisted of the attached script and a number 
of other^materials./ Six brands of pencils were used — two 15* pencils 
with #2 leads, two 15? pencils with #2^^1ekds, one 8* pencil with a 
#2 lead, and one 8<? pencil with a \#2J5 lead. All were national brands, 
painted yellow with pink erasers. The pencils were arremged in a dis- 
play box which showed the price of each^^encil; ^ix of each brand were 
included, three of which were sharpened aa^ three unsharpened. fifcrap 
paper, rulers^ and hand-held sharpeners were available to students. In 
addition , since most students in the fall seemed to overlook the $10 
limitation on the purchase of 'pencils, we included a. small sign with 
the figure "$10" and a picture of' a pencil among the materials -in front 
of students during their investigations, .(s'ee atta!ched illustration 
of Pencil Problem materials.) 

' ' ■ 

As indicated in the script, the last part of the instrument in- 
cluded a ^pdfestion about erasers. For this question, the sample pencil^ 
were removed from the tab^e and a box containing five types of -erasers ' 
and a "$10" sign were presented to studenJis^'^XAn illustration is 



. a1;tached.) - The five types were a. 356 soft -pink eraser , a. 35<^ hard 
red eraser / a* 3S* gum, eraser / a 25« ink and pencil eraser r and. a 25 
white plastic eraser; The purpose, of thi« gart of the instrument was 
.to pro^de. a crude indicator of J^e reliabirity of the xnstruineht ' ^. . ; 

^crpss different consiimer research problems. i ' 

. -■■ - •;; ..• . . ■ ,. ; . ■ • :■ ■ ■. ..-'i v 

Three stiidents at a time worked on the, problem. . Bach student ' 
weis asked to respond verbally to :the following .<juestiohSy over^the " 
course of the Pencil Problem session: ^ ^ . 0. 

• -• . .v^, ; ; , . ; . 'v. : -A . " . ' 

1. . ^•bb you' think kids can give (principal's name) good 
^/ . . advice . about which pencil to lDuy?" ; 

.. \ 2, . "How^will you. decide qn a pencil.?" . " - * 

3. "Which pencil do you think Xprincplpal'.s name) should 
•• . "^^^ ;l5uy/ cmd'why?" ■ 'l''-;- , ' 

• ' * ' "How wouM you try . to that 
lj ' ; . . the pencil ^jou- chose was the best ohe?*V . ' *. . 

. ' 5.- "What wou^ld, you. do tb ^decide which erais.er (he/she) 
' * : ' - ■ ■ should'' buy?;" " 

The ■ students * responses .were tape— recorde^ and they were also cisked to 
.indicate their' chqj^ce of; pencil^ and their reasons -in writing. Their 
written responses were given on an "Advice to Pj^incipal" sheet, (at- ^ 
t ached) ; - Student behavior . was observed as they worked with the pencils 
sthdthe observations recorded on an observer's sheet (also attached). 



The Pencil. Problem was aciministered in April and May 1977 to 336 



I 

students in the. four schools of th[e *1976-1*977 DSMES Student Study, 
Each student's verbal and written responses were coded on coding sheets 
(attached) designed to cover every common type of response. Observa- 
tions of student investigations were also coded. For the , verbal re- ■ 
sponses, two different coders listened to the tape recording for each 
^student and then independently coded the student's responses. Coded 
responses were entered into coinputer files. 



The ways in which the coded responses* were scored and analyzed 
are described in the report on the Student Study. Briefly, three out- 
comes were, examined: 

, • ' ■ ■■ ''. . -■ ■ ;. - . v "• ■ ■'- 

m . Total Number of Factors Considered. , . ^ . 

(How many of ; nine critical factors do students' motion, 
' at any -of several points in the Pencil Problem session. 
When di^ciissing t^e . quality and cost of pencils and 
erasers'Si^^ . ' ' 



Number of >qvestigations Carried Out 

C^ow many pencils do students "'write with, sharpen, amd 
erase with? . . pow^many other; investigations do they car- 
ry out?) 



-6- 



Response to "How Vfoxild /You Convince Oth^^r Students 
aaaat the Pencil You Chose was One?" ; 

(How many factors would they tell or show oth^r stu- - 
dents? Wpuid they have other students try them out? j ; 
Wbuld they ask other s^udents> opinions ?)^^^^. ^ • v 

This' is not the only possible meth^ of scpring^Pencil ProblMi ^ , 
sTesponses,^ however; -In fact, other methMs woiild pr<d>ably be inpa^^ 
appropriate for diagnostic lises of the instrument:. To date there has 
hot been an ade<paate opportunity to explore other scoring, -methods , sian^. 
users of the Pencil Problem wd:i3L need to determine ;^a method ^^ftiich is 
appropriate for^ their purposes. . ^ . . .1 . 



ERIC 




V- 



P^CIL PKDBI*EM SCRIPT.. 



Start tape. 

Write full names and 
ages' 6r% observer*^ 
sheet- . f 



Podnt: to $10. 00 sigh. 



HELLO. MV NAME 1$ (name) . . . WHAT ARE. YOUli NAMES 

AND , HOW OLD ARE YOU? ■ ^ / ■ . . ' . ■ : 

I AM INTERESTED IN . HOW STUDENTS SOLVE PROBLEMS, * HERE 
IS THE. PROBLEM WE WILL BE- TA3^I^ ABOUT. / .V 

YOUR PRINCIPAL, (■name>^ IS: GCOING TO SPEND -$ia. 00 'Td BUY 
pencils" FOR YOU AND OTHER STUDENTS TO USE. ."^HE PROBLEM 
IS WHICH PENCIL TO BtJY V I ' M ASKING ■ LOTS OF "KIDS TO'. LOOK 1 



OVER 



SOM^ 



PENCILS AND THEN SAY-. WHICH ' .ONE THEY Ttf INK ' 



. (principal's name) SHOULD .BUY AND WHY. .THE PENCIL HE/SHE' 
- • CHOOSES, WILL DEPEND- ON THE.ADVIC5 YOU GIVE .HIM/HER.. ; 

• - ^ BEFORE YOU TRY TO DECIDE ON A PENCIL, LET. ME ASK YOU 'THIS 

^^IDO YOU THINkI KIDS CAN^ GIVE (principal's n'^e) GOOD* ADVICE: 
..^ ^ . ■ ' "about WHICH^ENCIL TO BUY? I '^D LPKE TO HEAR/ WHAT EACH ' OF 

YOU THINK, SO .I'LL. ASK EACH OP'YdUr . r' - ' , 

■ . . . ■ . ' •> . ' . ■ ■ -■ ■ 

Ask each student. • DO YOU THINK KIDS CAN GIVE (principal Vs. name) GOOD ADVICE 



Ask* each student. 



Point to pencils. 
Point to prices. 



ABOUT WHICH PENCIL TO BUY", • (name) ? . 



WHY- DO' Y€U THINK 



SO? 



J 



# O.K., NOW, HOW WILL YOU DECIDE ON A PENCIL, tn^ame)^ 

NOW TAKE SOME TIME TO LOOK OVER THE PENCILS. THINK ^OUT 
WHAT (principal's name) WILL WANT TO KNOW. THEN. DECIDE 
WHICH PENCIL YOU THINK (principal's name) SHOULD' SPEND 
THE $10.00 ON, AND WHY? ^ . 
HERE ARE THE SIX KINDS HE/SHE IS THINKING ABOUT/ 
HERE IS HOW MUCH EACH KIND COSTS . / ' . 

YOU CAN^ USE ANY OF THE THINGS ON THE TABLE. YOU CAN WORK 
SEPARATELY OR TOGETHER--nfflICHEVER YOU WANT — BUT^ YOU DON'T ■ 
HAVE TO AGREE ON THE SAME PENCIL. IT'S O.K. TO TALK 'WHJLE 
YOU'RE WORKING WITH THE PENCILS. SO GO AHEAD AN© TAKE A 
FEW MINUTES TO DECIDE. 



^8- 



(Xlecbrd students* actions on observer's form. It's jbest If "^du seejn to be 
reading r .wrrting, etc^ Wait until the students seem to have finished exaxa-i ^ 
. ining the peircils before proceeding with the questions,) , " • 

■ ' . ' •.HAVE YOU 'FINISHED? . O.Rr. , NOW -I WANT, TO ASK BACT OT ^OU 

^ WHICH PENCIL YOU THINK (princxpal^^s^iaOT 



Ask each student. 



WHICH PENCIL DO YOU THINK (principal ' s. name) SHOULD BUY, 



Cname ) ? . 



WHY DID- YOU DECIDE ON THAT' ONE? 



•Giyfe ^Advice- to ■ NOVJ , WE HAVE TO LET (principal's KNOW WfflCH- PENCIL" 

Principal" sheetj^o > . : ' : > 

each ^st-udent. ' f ' YOU THItik HE/SHE SHOULD BUY -AND V WHY, . 3 

' ^. - . ' ■ ■ WRITE DOWN YOUR IDEAS- ON THESE sheets! 

^t ;^ 0M:op : part. ' - CHECK OFF. WHICH PEI^j^IL YOU CHOSE UP -HEI^. 

■Poiiit -to' b#:toro part.' THEN WRITE WHY^'sifOT CHOSE HERE.. . ' YQU D01Sr*"T 

• .-. ■ . HAVE TO - WRITE SENTENCElSr-^Y^ CAN JUST LIST YOUR. IDEASV. . ; 

V,^ - AND \0U DON'T HAVE TO FILL UP -THE WHpLE -PAGE. WRITE YOUR • 

/ : ' * ; / NAME-^AT 'THE BOTTOM. ■ , * . " ^ . 




,Collec;t Advice to 
Principal *• sheet. 
Ask each student. 



• NOW,. LET ME -.ASK YOU ; THIS:, .■ HOW ^- WOULD YOU TRY . TO CONVINCE 
OTHER STUDENTS THAT. THE , PENCIL YOU CHOSE. WAS ; THE BEST 
ONE,; (name)^*. 



Display erasers 
and $10.06 sign. 



• -NOW, ONE LAST THING, .HERE 



.'SjA 



DIFFERENT PROBLEM. 



Ask each student 




(Principal's- name) IS :,^Ij$O^INKIN^^ ABOUT BUYING $10.00 
WORTH OF ERASERS FOR STUDENTS TO USE. THESE ARE THE KINDS 
THAT HE/SHE IS THINKING. ABOUT. WE- DON ' T HAVE TIME NOW FOR 
YOU TO TRY TO CHOSE ONE, BUT IF YOU HAD AS MUCH TIME AS , 
YOU WANTED, 

- ■ ■ - ■ " ' . - -' • ■ 

WHAT WOULD YOU DO TO DECIDE WHICH. ERASER HE/SHE SHOULD 

BUY, Cname)? J ^ ' ■ ^ 

.O.K., YOUR IDEAS HAVE BEEN A BIG HELP. *YOU SHOULD. BE HEAR- 
.. ,ING ABOUT THE iPENCILS TN A COUPliE OF DAYS .... THANK YOU -VEI 






ERIC : • 
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V 





itiliricE 1^ PRIXCII^ 



Wldeli P«iiell To Buy: 

,□1 Dixon Ticonderoga #2 5/10. 

□ 2. " 

□a Gfleen-Rite #F 



)(jjcf)n Ticonderoga 12 

■i ■ ■■• ' ■ ■ ■ 



□ 4 
.□5. 



■£berhard Faber MONGOL IF 
Eberhard Faber' MON<^L ' f 2 



□ 6. Eberhard Faber #2 



inky-Bny Mat Pepeil: 
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PENCIL PROBLEM OBSERVER'S SHEET SCHOOL 



DATE 



TIME 



TAPE # 



NAME: 
AGE: 




Apprx.-No. written vith^ 
Apprx. No. erased with_ 
Apprx* No. sharpeneijj 

Wiat he/shfe wrote: 
-I 

^naine{s) ' 

lines 

ctures " 
.umbers. 

/ other writing (specify) 

V ■ • ■ 

.2.r^ 

Other investigations" 

. • calculating 

' ' - dropping. ^ 
■ . f leying pencils 
. * rae'^urlng (what & how) 



other (specify) 




Apprx/ No, written with^ 
Apprx. No. erased with_ 
ApprxJ No./ sharpened 




What he'/she wrote: 

i5_naine,(s); 

; lines * ' 

' pictures. , ^ 

^nuinbers . 

. ^other writing (speci€y) * 

Other "investigations" 

. . calculating 

dropping ' . . ^ ; - ' 

flexing pencils 

measuring (what & how) 



AGE: 



Apyrx." No. written -with 
Apipfx, So .. erased with 



other .(specify) 



Apprx. No.. sharpened ' ' 

What, he/she wrote > 

,name(s) 

lines ' 

pictures , * 

numbers . ' . , 

— - ♦ 

other^ writing (specify) 



Other "investigation^". 

• ■ "calculating 

^dropping . ■ 

Jlexing pencils^ \ 

' measuring (what & how) 



other (specify) 



TAPE 



_SroDE!JT*S SAME^ 



RESPONSES TO: DO YOU THINK KHS CAN GIVE; (principal) GOOD ADVICE jiBOOT WHICH PENCIL TO:BDI?- 



1. Y«5 J 

3. It depends/maybe 
"YES" REASONS ' 

ii. Kids use/vrite/vork vith pencils 

5. Kids heed pencils ' ^ . 

6. - Kids vaDt/like pencils 



7.-- Kids toiov ^fiout. pencils 



■8, Kids have morcAett^r ideas/are 
sniart 

% Kids can find out/thinlc of, . . 



10. Kids can t^ll/shov' principal 

11. Principal needs /would like advi ce 

12. ' Principal might get wrong pencil 



-13. Kids want 1^. give advice . "> 

— — \ ' n ■ V - ■ . 



^ik., .Soae. pencils. treak/f eel bad7ar.e 
too dark/light/thick/thin/etc. 



"NO" EEASQNS • . , ' 

1. ; Principal is buying so Be/she 
"should decide ' ^/^ ' ' 

2, ' Principal vould kilov about price 



3. Ki;ds vould choose* dlffereilt pencil 
them principal , , . 

I4. Kids vould choose- vrong pencil 

5. r'Kids' would spend too nmch 

6. Kids argue too much 



"IT- DEPENDS /MAYBE" REASOHS . - 
J. Depends on kids* .age- 
B. Depends on kids* experience 
. .9* Depends on how smart they are ' 
10, Depends on *the pencil > ' 




OTHER REASONS 



.I." Different Idds like different .pencils v 

2« If principal giTes then to us, iie voi^^t . 
have^to buy thett ; • ■", • 

3. • I have seme of the pencils .p ••, 



1*. don't know/hot suife'r etc/ 
5. |UninteUigible) , , . 



» 



■'1. 



- " ■ iSi > > * ' ■ STUDENT'S NAME 

RESPONSES TO: HOW WILL YOU DECIDE ON A PENClJ? " 



s ■ 



,1a I'd choose/pick/take/decide on the 
«r one that.'.. 

«• _ . ' . 
(Ho vert) The one that^/If it/Hov 

3. It .depends on..* * 

k<. Vi seie/find/look fof one^that/ 

5. I'd see/flnid-out if it;.. ' 

6. IM sec/find out how it..../ 

7. I'd see/find \out/lodt at what it... 

8. i like...: - • . 



9. , I'd vrite with then/use then 

l£; I'd shiarpen tfem\ ' 
• ... 

11. , I'd -erase- with then' 

12. ,1'd measure theo- 

13. I^d headlfthem * * , , 
it. I'd hold them/feel than ■' 
15*. I'd ask other Mds.: . - 
l€. I'd try' then but/test them 
17. I'^d Idok them over 



SECOND PHRASE 



5 



1. Writes/vorks . * ^ ' 

' . jfi- 

2. Wfites/vorks well/good/hestynice- 

3. Writes dar]^ , . ^ 
If. Writes light ' ^ . 

5. Writes medium/not too light or dark' 

6. Writes smooth/doelsa't scratch 

7. Writes easy *v 

8. Writes 'neat ' ^ ^ 

I f 

9. Writes small /thin- * 



10. , Sharpens ' ' 

11. Sharpens vell/good/hest 

.12'. Sharpens vithout.l^reaking lead 

■ ill' ■ , 

13. Sharpens without cheving up vood^ 



ih. Erases 

15. Ilrases vell/good/hest/has a good 
eraser 



l6. ^as a hfg 'eraser • ^ ' 
^T. Erases clean 

,8. . Erases without crunhs 
19. Erases without tearing' pat>er 



20. Costs ' , 

21. Costs less/cheapest/lovest 

22. Costs morei/iexpensive 

23. Costs 8^ 
Costs I?'*' 




MORE SECOND PHRASES 

1. Has a good lead 

2. Has a' #2 lead ' .■ 

t4 ■ 

3. Has a strong lead /won't break 



h. Is good/right s'ize 

5. Is long/big 

6. -Is. thick/fat 

T. .'Is- thin/skinny' 



8. Pencil is string/won't break/won't bend 

9. Has good wood 
10." 'Pencil lasts long 



11. .Feelis good/is comfortable * - ' 

12. Is pretty /lopks gocS/has good color 

13. Is sharp " ^' ' ■ 
lit. Has good name 
15*. Is good quality 
16., Is best . , 



DTHER RESPONSES 



17. (Names a specifi^c pencil) 
L8. I don't know/aot sure'; etc. 



L9. ^Unintelligible)^ 



Ay 



to 



RESPONSES TO 



STUDEBT'S HAHB 



WHICH PEKIL DO YOU THIHK (princ^) ^SHOUU) BUY, AMD WHY? 



FEHu'Ili CHOSES , 

1. Dixon Ticonderoga #2 5/10 (aedium)^ 

2. Dijcon Ticonderoga #2 (soft)^ 
. .3.-" Grccn^Rite #F ; ' 

■. V/'Eberhard/PaW MONGOL #F 
■■■*5.. Ebcrbard ^*ber^ MONGOL *2 . 
jS. Ebdrhard^aber #2 / 

' 7. Writes/works vell/goojj:/better/best/ 
nice • 

.8, Dravs.wfeU/good/better/best/nice^ 

9. Writes aeat 

i 

10.' Writes easy v , 

* ■ . . * '' 

U. Writes smooth. * * : 

\. ■ ' ^ ^ . '■■ 

,12. .Doesn't scratch/dig into paper 

13. Writes, snall/thin/skinny . 

Jl*.^ Writes thick/fat' ' * . ' 
:l5. Writes light/not dark * 

l6. Writes dark/not ,light . ' 
,17. Writes heavy' 

■ 18; Writes medium/not too light or dark 
^19. Doesn't .smear ^ . 



SHARPENING 

20. Sharpens veil/good/better 

21. Gets a sjiarp point 

22. Lead doesn't break" 

"^3. Lead^doesn't f^ll out ' 
^1*. Wood doesn't crpble/get cheved uf 



ERASING ' 

1. Erases 

2. Eraser 

3. Eraser 
h . Il-aser 
5. Eraser 
6^ Eraser 

7. Eraser 

8. Eraser 



vell/gocd/clean/good eraser 
is stroiig ; ; , ^ 

is big : 

is RUt^in yellj' 




doesji't come but 
doesn't vear dovn. 
doesn't leave crumbs ■ 
doesn't rip paperVmake holes 



COST 

9 . Is, , low/ cheap/inexpensive/less 

10. Is high/expensive • . . 

11. Is ■ 

12. Is 15^. 

13. Is a good 'deal 



LEAD 

lli. Has good/best lead . , ■ • - 

15. Has .a #2 lead 

16., Has strong lead/lead doesri't break ' 
17. Has hard lead/not too soft 
18 f Has soft lead/not too^ard 
i9. Lead lasts long/von 't^,vear^doim*fas:t 




_ — ^ 

■i , > . 



• .84 



SIZE' ' •• - 

' -ij . • • , "'' '* ■ '« v" ■ ■■■ 
1. Is.good/right';8iie.- ^ : . . , \ ' 

'2. Is lonjg/big \ 

3/;Is short/small '• '* 

i*. Is: thick/fat- ; , ^ . : •' • 

. 5- Is^thin/skinny • , . 

OTHER DESCRIPTIONS ABOUT PENCIL ^ " . ^ , . 

6. Pencil is strongAvon^t break/von't tend^ 

7. Pencil -lasts long/'von't be used.up qai'ckly 

8. Pencil has . good vDod ' 

9. Feels good/is comfortable . . 

10. Is. pretty AoQks goo(J/has good color ^ , . 
.U. TX sharp^. ■ ' ' /■ ^^''^:. 

i^.*'.Has goodnaae, \\ ^< " ' v 

13., Is good quality/good/great . ' 
11*. (Names bad things* about other pencils) 



'i^ OTHER RESPONSES / • . 

15. I'm used to" it/have used it a lot 

16. Hike it ' 



•17^. I don't knoy/not sure, >tc. 
l8:CUnintelligil5le) ' ' 



STUDENT'S Sim 



' RESPOHSES TO: HOW WOULD YOU TRY TO COPIUCE OTHER STODEMTS 'MT'tSI! PENCIL 'YOU CHOSE WAS THE BEST. ONE? 



FIHST PHRASE ' 

1. I'd teli thm/Pd say... 
i. I'd show then... 
3^ I»d compare... ' . . • 
' No first phrase 



10 



'a 



SECOND PHRASE 



1. (about hov it writes) 

2. (about .hov it sharpens) * 

3. (about .^v it erases/about eraser) / 
(about its cost) 

5'. (about ..its lead) . 

6. (about its size) ' . ' 

7. (about its feel) 

3. (about pencil '.breaking) 

9* \ (about hov long, it last's)- - 

10. (about hov gooa it is) 

U. (about 4i-ferent p«ncils--nothing. 
. specific ■ : ■ . 



12: (about 



.) 



5. I'd let* them..,. ' * , ; 

6. I'd^Mave then/ask then to/tell l|hen 
to... 



^■f. I'd ask then... 
16. I'd see-.. 




13. Wpite/use/drav vi-th it 
ih. Erase with it 

15. Try it'/test.lt * 

^ t - . 

16. Look at it- 

IT. Vote . 

18. Compare 

19. Deciii 




20. If/hov'they like^^t/vhat they think 

. of it 
» , 

21. ' What they like/think ■ 



■*lv"I'd try to convince '.the 
.*2; J'd Ask^vbich they' likticf"- 
,*3,. I'd compare 
*l*/rv6uldD't' mind. . 



OTHER RESPOHSES 



5. ILjiQn't'knov/nbt Sure,etc. 

6. (Unintelligible) • 



*9. ^If they didn't :lfke it 



. ■ ■ ■ 



STUDaH'S EAME 



'1 



RESPOHSES TOi /WHAT WOILD YOU'-DO TO DECIDE WHICS 



^he/she should buy?' 
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• • I ' d . chooae/pick/take/decide . (jn thei 
' 2. (Ho verb)' The ont' tha^yilf it/Hov 




J./' It depend^ on.... 



X* rd see/find/look for oinVtMt/ 
. ' which... . .;■ ' 

5; I'd see/ find ;oirt|if itj 

'6, L'd' see/find out how ' 

7. I'd '.see/f{nd out/looic at 
vB^it.lv • ' 

9. ' .r like.:. 



9. I'd, erase with them/use them 

10. I'd write. agd erase with them* 

U.' I'd* measure them • ^« 

,12.- I'd behd them 

13. ♦ I'd hold tW/feel them 

ll;. I'd ask Other kids... , 

15 . Vi .tiy them' out /test them 

l6. I'd look them . over 



ll* 



SECOm) PHRASE 

1. Erases /works - 

2. Erases/works ' veil/better A>est 

3. Doesn't smear /erases clean 

1*. Doesfi't break/cnimhle/f all apart ^ 

5* Doesn't tear pi^r/make holes 

6. Lasts long ■ ■ H 



... ■ 

^ — — \ 

T. Is big;{long- ' ' . 

8- Is'strpng \ ■ 
9. Is hard 

10. Doesn't bend. too much' ' 



11. Costs . ^' 

12r Costs less/cheapest/lovest 
13. Costs more/e;cpcnsive 
lli. Costs 25^ 
15- .Costs 35^ ■ 
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SECOND 
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"1. Feels good/is ccmfortable '/ ...^ 

2. Is pretty /looks good/has good color / 

3. Has good name 

It. Is good quality. ' ' 

5. Is best ■ >■■ ■• 



OTHER RESPONSES 



'6. ''^(Hanes a "specific eraser) 

7. (Hanes a specific eraser and 
gives reasons) / 

8. (Barnes "bad things about other erasers) 



9. .1 don't -knov/nQt sure-,' etc. 



10. (Unintelligible) 



. PENCIL choseh '" • 

• "'l. Dixon. Ticonderogii #2 5/10 (mediim) 

2. Dixon Ticonderoga #2 (soft) * . 

• 31 Green-Rite ' \. j.' 
I*.. Eberhard Faber MOBGQL ifF" 

• : ' 5* . E^rhard Faber MONtfOL 

■ 6. ' Eberhard Faber #2 *: 7 

WRITING ^ 

. 7;. Writes/vorks veil /good Ajetter/l^est/ 
• -riice • • ^ * 

; .8.' Draws vell/good/bett^r/best/nlce / 
. . "9v Vsites neat' ' - ' 

• 10. '.Writes .easy . . , ' • • 
^ 11. Writes 3rooffth ■ " 
:*12. "DoeSn \p "sicrfttch/di^T ^in-to « paper 
. I3. Wntci smali/thin/skinny 

iM.. Writes thicX/fat 
15- Writes light/not dark 
ri6. Writes' dari/not lic:ht 
IT* './rites heavy ■ 

18. Vfrites mediun/r.ot too li^ht or dark 
19- Doesn't smear 



" SHARPENING " 

20. 'Sharpens veil /good /better 

21. Gets a sharp point 

22. Lead doesn't break ,' 

23. Lead doesn'^t fall out ' ' ' 

'2^; Wood doesn't crujnbl'e/get choked up" 



-RSSP0HSSS:.t6; advice to PRINCIPAL SHEET 



ERASING 
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1, Erases vell/good/clean/good eraser 

;2, Eraser is strong 

3. Srase^ is big.' 

' i*, Eraseris put in veil * * 

5- Eraser doesn^t come out 
6, Eraser dfoesn't vear down , ./ 

7- Eraser dopsn't leave crumbs ' ^ ■ ,4 

8. Eraser doesn^t rijD paper/make holes 



■COST 

■ ■<.■.', 

9. Is. I0V/.C heap/ inexpensive/less,, 

■' ^■ 

10. Is high/expensive 

11. Is ' 

12. Is I5i 

13. Is a good. deal . 



LEAD 

lii. Has good/hest lead 
15-. Has a ^'S lead ■ 

iG. Has a strong lead/lead doesn't treak 

17. Has hard lead/ not too soft 

18. Has soft lead/not too hard 

19. Lead l^sts long/vqn't vear down fast 



SIZfe 

' 1 , Is good/^ight size 

• 2. Is long/big 
,,, 3. Is short/small \' 
h. "Is thick/fat ; i ; 
'..•:5.'Is thin/skinny \\ ■ ' ' 

OTHSRDESCRIPTIONS-ABOUT PENCILS 

.6. Pencil' is'^tfong/von't break/von't bend 

. '7' Pencil lasts long/von'g be used up ^ 
quickly . * ^, ; 

8 . ''Pencil has good vobd ■■ " 

9. Feels good/is coniforta^le 

10. ^' Is pretty/looks gopd/^s go^' color . 

11. Is sharp' _ J ^ 
■12. Has good name 

13 i Is good •quality/gisod/great 



Ih, (Names bad tf 



but other pencils) 



. OTHER RESPONSES . , • V 

15. used to it /have used' it a lot 



16. I like i,t 



17. I don't knov/not' sure, etc, 

18. " (Unintelligible) 



V . ■ 
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ADMINISTRATOR'S OBSERVATJOITS 



i 



student's name: 



^ WHAT STUDENT WROTE 
:1. .Naines.. 

,2., -Lines', /' 
■'.3» "Pictures.^ 
IWabers 

■5.' Scribbles, sqrfgglei^'' 

7**Vtters - 
'■•3, Shadings 
9- Otjier writing 

I 
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INVESTIGATIONS 



["ONS i 



20 



■I... .-Measured' length's of pencils 
vith ruler' * ; ^ ■" 

Z. Compared lengths of .p.eincils' 
by sight . ■ 

3. Compared sizes .ofverasers 

l^. "Weighed" pencils, by, feel 

5-'%0bserved time to sharpen 

6. Picked up and looted at/ 
examined details of . 
construction' 

7- Pressed point to test lead 

8, -Breaks lead 

9.. ; Smeared writing 

10. Handled erasers ■ ■ 

^ 11. Calculating ■ • 

12* Dropping 

■ 13- Flexing pencils 



V. 
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'22. ■ Vritten v/_ 



'i3.* Erased v/ 



I 



21^. Sharpened J_ 
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BACKGROUND. 



In deciding which student attitude:^^gre most important fbt- the- ' • " 
1976-1977 research, the USMES staff ^TevxesnM n , 
used in previous rese^j^cfe " Ex:i^t;ihg pii^ ; 
.reviewed' earlier V^^.^ by the ^ USMES staff >and. by. tji^^ostbn-^nive^^ 
V USMES evaluation stikff, but nl^ -published; instruxmgnt hajd^been found, 
•which corcred the ^'articular Attitudes most ^V^^ to\ji® USMES type 

of xeadL problem solving. -Howevfer, s^me of the items use^^ by the USMES 
staf fi in trie 1972-] 973 district implementation experiment in Lansing, 
Michigan and by the Boston University; group in their 1974-197^Bvalua- 




tion program sugge: 



ted topics which might be of interest. 



Eventually, the USMES staff agreed •'that four types of attitudes 
should be studied^ in. the 1976-197^ .research: (1) attitudes towards 
working on real- problems and producing effective solutions; (2) / 
attitydes toward group iritBraction;^ (3) attitudes towards specific 
problem-splying activities (like working with numbers surveying, 
and writing); and (4) attitudes toward USMES (for students with ^ . 
: USMES experience)* These categories of questions were thought to 
cover the main attitudinal goals of the USMES program and also the 
key attitudes which might affect the students'^ motiration ^ get 
fully involved in USMES activities. ■ r j ' 

. The USMES research staf^, in cons€iltation" with others: in the ■ 
USMES staf^ and with other evaluation specialists -at EDC, developed 
' a threerpage questionnaire? covering the' above topics, during- the 

spring" and summer of 19f?6 . : A. draft version of . the questionnaire which 
was c;alled ,the QuestioiMiaire on Attitudes towards Reail. ProbJ.em Solving 
(QARPS) was sent for revxew to experienced USMES ^teachers and admin- - 
istra'tbrs in . Iowa/ Texas /Nand Minnesota. Their suggestions were, in- 
corporated in'the fall version of .the QARPS. . 
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^THE TALfe QARPS 



JARPS consisted of two parts: (1) twenty 
lon-USMES Students oh attitudes toward 



- The fall version of the 
questions for botih USMES and 

working on" real problems , croybp work , .and specific re a^ problem solv- 
ing activities; and (2) ^mv^n questions for experienced USMES- students 
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on: attituSw toward v.USMES^ T?he Vf font pajge ' of : the- questionr^aire con- 
tained space ^for i^ntifiyation* iivformation and directions .for stu- , 
dents. A sample* of the ''i^v^" letter which^ accompanied the fall 
•QARPS: is. attached. . • ' >. 



. • ^ In the fall, thil^yf three classl^-. (?49 stu<^rits) ; ijri grades-! 
thrdragh 6. of the four; schools in the Student .Study received the QABPS. 
Th/2se ):sila^^es included most of the cleisses doing US3MES this year- in v 
:'these sclfdolSjr^ eilthough_- very* few classes Ka^ begun . USMES \TOrk a^ 
time of . the fall QARPS administration;- in mid-;p^^ 
pther classes '^''^ the saijie* grade levels in the^scnools^^^^ 
students in eadljl^'ass . filled '/ ■ , -^^^ / ' ' - 

.If ' 'Inspection "of the fall results' showed that sqme questidns .wiere 
unclear, especially to yoxinger students. These were reworded^ in the 
spri>ig version. Alsb there appeared. to be some unnecessary redun- ^ 
daxicyss^TC some omitted topics." To make up for this "some =^of the fall 
questions were replaced with new ones in the 'spring QARPS.J 

• -THE SPRING QAKPS ' _ • •" ' ' 

• ' The spriiig version of the -QARPS (attached) also contained twenty . 
questions cgi attitudes toward real problem solvinl^. Ele:^n pf . these •* 
questions contained minor changes of wording, and two' were sats^i^ 
tions. for questions used in the fall.- The spring version ajfeo con- . . 
taineA nine questions on -attitxades towkrd USMES^ and-a cljljpKlist on 
USMEsI activities; this section was intended for^^SMEs^i^udents; only. 
One o£ these questions was a Vinor rewording of a fall qtiestion, and 
three Jwere new to the sprihg version.. No coyer letter was ..prepared 
for tile spring version / since "'it was administered entirely by the 
USMES research staff. ^:>r 

. In the spring, thirty-seven classes (1 ,011 students^) in grades 
2 through 6 filled out the QARPS. " The sample of classes was chosen 
to assure coverage of key variables in the Student Study. . 

. " • ' RELIABILITY AND VALIDITY * * - . 

*- . We have not been ^le to conduct technical studies of the instru-- 
ment, such aS; formal ^reliability studies or systematic development of 
.norms. However r we have examined several kinds of evidence cibout the 
qualities of" the. instrument. None -of these types of evi<ience are as . 
strong as we wish by ther^elves, but they are reasonably persuasive 
when' consid^^d together. They include the ob&a^red consistency of 
responses on many items from fall to spring •:.and' tne*»£airly. satisfying 
inter-rcorrelations of responses'*"^5o"^f ferent questions, roughly, as we 
had predicted. \ 

The positive reactions received ^om teachers others' about 

the appropriateness of QARPS questions^rovides sdme reassurance about 
the content validity^ of the instrument.- Responses to some of the'.. 
QARPS questions also show expected trends by age and school . _ - 



"J " /scoring AND analysis' 

. " • * ' / ' *** ■ ' ■ • ■ 

'"^ . - For the Student Study analysis , the twenty questions on ieal 
problem solving were combined into three .scales corresponding to the - , 
three types of attitudes the ^ instrument was designed to measure i ' . 

Attitudes Toward Work on Real Prdblems^^^d^ PxLQ^cin^ Effectrlve ^golutions 

V * (87 items) : : / • . / 

.".•«. ■" . , "/ ■ . ^ , . . ■ ■ ". - ■ « . " ■ • 

>>; V "If kids shoi^: that a change is needed, then grown^ . 

ups will go along." . :^ ; 



• : -."JC figtire out what I'm going to do before I start 

s^omething.'" ^ / ' ^ 

• "I<ids usually can't come up. with, good ways to solve- 
' problems around school. " (Codfed in reverse , e:g., 

. "Disagree a lot" coded highest-. J " *^ ; V " 

.. \ ^- ■ , ■ ' ' ■- 

m "Usu^ly -there is 3^s% one way to solve , a i;eal life • 
. - . p^roblem. " (.Coded in 'reverse. ) . ' vJ . - -. 

; V"I* 'like ' to ciake/ things better «i:ound "school. " - . 

j^'^^'I' think 'l am good at solving real life problems." . 

"As. soon as I think of one-way to solve a problem, ' - 
I don't think, about .other ways." (Coded in reverse.) 

" : ■ V ^ - ■ ■ .' .. --. .^^ >P 

"I like working on problems that have- mo lie/ than one 

answer,",. * ' ' * 

» . ■ ■ ^ - 

Attitudes Toward jflSyoup Interaction 

• . (7 items) / . . 

"I like "to work in small groups with other students . " 

"I like to help decide what we do in class.*" 

"I like to wojrk by myself." (Coded ^ reverse 

"If - I think, othier: kids won't like my idea, then. I 
keep, qud^t about itV>^x>^CbdedJLil^e^erse . ) 




^0. "JC- like to listea to other •studeMS talk about their 
. ideas." . : 




• . "I .lfcike to ask que^ions . " ' \^ ^ ^ 

- •- - v" I like, to talk to other student sNabout my ideas." 
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Attitudes Toward Specdfic 
Problem-Solving. Activities 
• . ' (5 items) 



■■{ 



"1 like to write about my ideals." 



A- 



"I 'like to measure things, when it Tielps-'' solve r«al . 
life problems. T ^' \ " " " ^ \ - 

"I like tojaake clJ^J^s Ind graphs that j|how thing^^ 
I' found out." • ;osy 



I- 



like to use numbers to ^solve real lif^ problems: " 



"I like 
think. 



e il^^a^ surveys ^o firid'^ x>nt what *^ther , pe:6ple * 



■ These three scfailes- do hot ^trepres.ent th'e\ctolv way to analyze \ 
QARPS respite In. fact-; for informal diagnbi^Lc puri^ may be 

more appropriate to >^ook at^esp^mses item item for individual 
studenfes aCnd, using meansv/for thfe class as a whole. To date there' 
has not been tiiw to^^exj^lCTie other scoring methq^ cuid .QARPS users 
will >/ant to investigate* i|^er procedures for themselves. 

^ Responses to the ten questions about the USMES program - (on the 
back page of the; QARPS) 'may b^ examined item by item' to teveal how 
individual. students or the class as a whole i^ experieiicing USMES. 
Responses to past administrations of similar item^ ar^ reported on 
in Chapter V of the f ourth edition of the USMES -Guide '^and may be * 

used for comparison purposes. -. -r 
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COVER LETTER FOR FALL QARPS ;„ 



TO: teachers and. othejrs-.who plan to use the Questionnaire o^ Attitudes 



toward Real Problem Solving (QARPS)- 
FROM: The. USMES- resear<^h staff 



" The purpose of the QARPS Is to asse^^s some attitudes that tnay be ^ 
related to students* performance In solving "real problems". IX is 
intendfti^for use both by teachers who want to examine the ,^in>gress of 
tWlr owi)'^ classes, and by researchers or curriculum Supervisors who are . 
. studying the impact of pre grama to teach real problem solving. We '•expect 
tharfr. stildents in grades ^^-8- vl 11 be able, to read and fill but: the QARPS 
foda on . their ovn^^-'gCu geL ' its in grade 3 or below may be able to fill out 
their oiim respo^es if their teaser reads each item out loud. 

The first twenty items (pages 1 and 2) should;be ap^oprlate for 
both ^USMES students and students who have, done' no USMES work- The last • 
section of the QARPS (page 3). applies to USMES students only and should 
,be ignored when other students are usihg the questionnaire. 

•• ■ ■ ■■■ ■•■ ■ • 

Students should: understand that the word '*^Mpb^ein" as used in .this^ 
Jluestlonnalre refers to .^sractical, open-ended' jWpig*^ — like trying to 
/improve 'some si tuft ion in^ the school or_^oiranDTtitto-not word problems or 
workbook exercises., , For this reason, ije suggest^hat you read, the ' ^ 
"Directions for students'V on "the cover of the QARPS out -loud before the 
students begim to fill out the questionnaire.- - \r 

. ■ * . ' . ' , . ■ •' ■ 

- Depending on your purposes in using the. QARPS, you. may- want. to--have • 

the students provide more or- less bacllground information than is indicated" 

on the cover sheet. The infbrmation that-is^ requtf«ted wil'l be. most useful 

for USMES *s research purposes, ' and we hope you will Include it in any data 

that you can send to^^j^s Csee b'clow) We >have not asked for student^' names 

since it of teyv seems, best on this kind of questionnaire to let students 

remain ahohyTpbus. However, names o^r identification numbers can be added 

in a blank part . of the cover -sheet , if yoii prefer. , 

" ^Comments and suggestions about this '/final draft" are wijlcome^ In, 
addition* we .would like to ^ee what results you obtain so w^j^an assess' 
the usefulness of the questionnaire. 'We will be happy "to comput^ tallies 
and per'centaiges for you (as long as our funding permits). Please send 
comments, c(>mpleted questionnaires, or results you tally yourself, to Dan- 
Cooper or Carolyn Arbetter ac the USMES office at EOC. - " * 
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Questionnaire on Attitudes toward Real Problem Sd I ving .• 



Your name : 



Your teachei/'s neune: 
Did ycfti do tfSMES last year? 



DIRECTIONS FOR STUDENTS:. ■ 

* VJe would like to know, what- you think jbbout some "things around 
school. Your answers will be used to try /to make some sch''* •I work 
more interesting. 



jCnside there are some questions that/ we yould* xike, ypu to 
answer. This is not a. test, anS there ai/e nq " right answers. 
VJe just want . to know how you ^eel. 

Each question asks if you agree or disagree with" something 
\other students have said. Here is aiij|bxamp.le f or "you to try.' 
»^lease make a mark ^through the circle ^hat goes with your choice: 

I like to write stories. ^ 
Q Agree a lot 
' O Agree a little 

- , ' . . O Disagree, a little 

O Disagree a lot 




It may be har€f to make up your mind about some questions. 
Just. pick the answer which is closest to how ^youv feel . .Don't 
spenij too long on any one question. * " - '■ -^_) 

Several questions are about- ''solving problems." They donVt 
mean math problems or problenrs in work 'books. They mean real li;|e 
■ problems',^ like trying to' make something you. do in ^school safer or 
more ' fun* ^ . - . ^ 



If you have any question^, you should ask -them* now. . 
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1. If kids show that a change is 
needed, then grownups will, go 
along . 

O Agree a lot 
O Agree a -little 
O Disagree a little. \ , 
^ ^ - Q Disagree a lot ^ 



2-1 like to work in ismall,5gti?Aips 
with other students, "* ^ •■ 

O Agree a lot . 
' O Agree a little ' 
O Disagree a little 
O Disagree a lot ^ 

3. I like to write about my ideas. 
O Agree a lot 
Q Agree a littls» 
O Disagree a little 
' Q Disagree a lot 



4. I figure out wha.t I'm going to 
. do before I start something - 
O Agree a lot 
Q Agree a little ^ 
O Disagree a 'little 
Q Disagree a lot ^ 



5- Kids usually can't come up with 
good ways to solve problems 
around school . 

Q Agree a lot- 
Q Agree a little 
Q Disagree a little"* 
Q Disagree a lot 



6. I l-ik^ to help decide what we 
do in class. 

Q Agree a lot 
O Agree a Little 
Q Disagree a little 
Q Disagree a lot 



7. Usually there is just one way 
to solve a real . life problem. 
Q^Agree a lot , 
Q Agree a little, 
Q Disagree a little* 
O Disagree a lot 



8. I like to measure things when 
it helps -solve real life 
problems. 

O Agree 'a lot s 
J O Agree -a little 
■ " O Disagree a little 
O Disagree a lot 

9- I like to work by myself. 

Q Agree a lot 



Q Agree a little 
Q Disagree a little 
Q Disagree a lot \ 



10. I like to make things better 
around .school. 

O Agree a lot 
Q Agree a little 
O Disagree a Iri-ttle 
- ' O Disagree a lot 
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11. If I" think other kids won't like 
my idea/, then I keep quiet about 

/ it^ • ' 

Q Agree a lot 
O ^9^^^ ^ Tittle 
. O Disagree a littlje 
Di-sagree'a lot 

12. I 'like to ask questions. 

Q. Agree *a lot 
■Q Agree a little 
Q Disagi'ee a little 
Q Disagree a Ic^t ^ 



13. T 

like to make charts and graphs - 
- that show things I found out. 
Q Agree a lot 

Agree a little 
Q Disagree a' little 
/ O Disagree a. lot 

14. I think I am good at solving^'real 
life problems. 

Q Agree a lot- 
O ^9^^^ ^ little 
Q Disagree a little 
Q Disagree a lot 



15. 1 like to talk to other students 
'"^ about my ideas. 

Q Agree , a lot 
Q Agree a little 
Q Disagree a little 
Q Disagree a lot 



16. I like to use numbers to solve . 
real life' probl^s. 

Q Agree a lot • 
- . Q Agree a-little 

" Q Disagree a little 
Q'^Disagree a lot > 



17. As soon as I think of one way 
to solve a problem, -I dbn't ^* 
think about other ways. 

Q Agree a lot' ^ 
Q Agree a little 

Disagree a little 
Q Di^sagree ' a - lot 

18. I like to, listen to other 
studf^nts talk about their 
ideas-. 

" O ^9^^^ ^ lo't 
. Q Agree a little 

Q Disagree a little 
'* Q Disagree a lot 

19/ I like to do surveys to find \ 
^ out what othej? people think. , 
' Q Agree -a lo^ 
. -Q Agree a little' 
Q Disagree a little 
, Disagree a 'lot 

20. I like working on problems that- 
have more than one answer. 
QAgree a lot 
O Agree a little 
O^^isagree a little ^ 
O^^isagree a lot 
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21. ' I think USMES worj^ 

is fun. 
O Agree a lot 
Q Agree a little 
Q Disagree a little 
Q Disagree a lot 

22. " i think USMES' work 

is boring. 
.Q Agree a lot 
Q Agree a little 
Q Disagree a little 
Q. Disagree a lot 

23/ Doin'g USMES is hard 
work. 
O Agree a lot 
Q Agree a little 
Q-Disag^ee a little 
Q Disagree a lot 



24. I fion * t know why we 
do some things in 
USMES. , - 

Q Agree a lot • 
Q Agree a little 
O Disagree a little 
O Disagree a lot 

25. Doing USMES makes jne 
think. ' • 

Q Agree a lot 
Q Agree a little . 
Q Disagree a little 
Disagree a lot .• 

26. ^ I think USMES work is 

confusing . 

Q Agree a lot 
' Q Agree a little 
^ >Q Disagree' a little 
Q Disagree a lot 



27. I think USMES work is 
important.' 

O Agree a lot 
O Agree a little 
O Disagree a little 
Q disagree a lot 
i - ■ \ ■ ; 

28. In USMES it-'s hard to 
decide what to do next* 

Q A^ree lot ' 
Q Agree a little 
Q Disagree a little 
Q Disagree a lot 



29. I. would like to qS?" 

' more USMES . . * 4c . 

A^ree a lot \ . ; • 
O Agree a little v 
O Disagree a ^little 
Q Disagree a lot 



30. When we do USMES, usually do these things: 
(Put a check next to the ones- you usually do.) 



mea'sure things 

do su rvey s of pe o • ^1 e 

build things 

ask questions 

write about things we 
find out 



work in small groups ■ 
make graphs or. charts 
do math 

help d^ide 'What we d(? 
tialk about; my ideas 
work by myself 



edc 



Thank you. > 



ERIC 



ERIC 



